LLM Observability:
The Complete 2025 Guide

1. Why LLM Observability Matters

The shift from traditional software to LLM-powered applications introduces an entirely new class of production problems. A REST API either returns a 200 or it doesn't. An LLM call can return a 200 with confidently wrong information, a hallucinated citation, a response in the wrong language, or a 4,000-token essay when a two-sentence answer was needed. Classical monitoring — uptime checks and error rate dashboards — cannot catch any of this.

LLM observability is the discipline of instrumenting AI systems so that every inference call, reasoning chain, tool invocation, and retrieval step is visible, queryable, and evaluable. It draws from the four pillars of observability in distributed systems — traces, logs, metrics, and evaluations — and extends them for the unique characteristics of generative AI.

The Real-World Consequences of Blind LLM Deployments

Teams that ship LLM features without observability typically discover problems through user complaints, not internal detection. Common failure modes include: silent context-window overflows that truncate the prompt and degrade response quality, prompt injection attacks that go undetected because there are no logs to audit, runaway token usage from a single buggy agent loop that inflates the monthly API bill by 10x, and gradual model drift when a provider silently updates a model version.

With proper observability, you can catch a p99 latency spike within minutes, attribute it to a specific retrieval step, identify the document set causing the bloat, and push a fix — all before your SLA is breached.

2. Key Metrics to Track for LLM Applications

Not all metrics are created equal. Some are vanity metrics that look good on a dashboard but don't predict user experience. Here are the metrics that actually matter, organized by the problem they diagnose.

Latency

Track time-to-first-token (TTFT) and total generation time separately. TTFT determines perceived responsiveness in streaming UIs — users notice when it exceeds ~500ms. Total generation time matters for batch pipelines. Always track p50, p95, and p99 to catch tail latency. A high p99 with a low p50 usually points to occasional context-length overflows or model rate-limiting causing queuing delays.

Token Usage

Input and output tokens are the primary cost driver for virtually every hosted model. Log tokens at the span level so you can identify which prompt template, retrieval configuration, or agent step is responsible for token bloat. Set alerts when a single trace exceeds a configurable token budget — this often catches infinite agent loops before they drain your API credits.

Cost-Per-Query

Calculate real-time cost from token counts and current model pricing. Store it as a numeric field on every trace so you can query "show me the 100 most expensive queries this week." This is the single most actionable metric for LLM cost optimization because it lets you correlate high cost with specific user segments, use cases, or prompt templates.

Error Rates

Track distinct error categories: API errors (rate limits, timeouts, 5xx from the provider), context length exceeded errors, JSON decode failures (for structured output), and tool call failures. Rate-limit errors should trigger an alert at >1% of requests — they indicate under-provisioned throughput or a runaway process. Context-length errors indicate a prompt engineering issue.

Hallucination Rate

This is the hardest metric to measure but often the most important. Automated approaches include: running a second LLM call as a judge to verify the answer against source documents (faithfulness scoring), checking whether all claims in the output are grounded in the provided context, and comparing answers to a golden dataset of known-correct responses. Even scoring 5% of your production traffic gives statistically meaningful hallucination rate trends over time.

Retrieval Quality (for RAG Systems)

For retrieval-augmented generation pipelines, measure hit rate (did the retrieved chunks contain the answer?), MRR (mean reciprocal rank of the correct chunk), and context relevance (are retrieved chunks relevant to the query?). Poor retrieval quality is the leading cause of hallucinations in RAG systems and will not be visible without dedicated retrieval tracing.

User Satisfaction

Capture explicit feedback — thumbs up/down, star ratings, or "flag as incorrect" signals — and link them to the exact trace that generated that response. This creates a labeled dataset of good and bad outputs that can drive evaluation fine-tuning and surfaces systematic failure modes that automated evaluators miss.

3. Tool Comparison: Langfuse vs LangSmith vs W&B vs Arize vs OpenTelemetry

The LLM observability tool market exploded between 2023 and 2025. Here is an honest, side-by-side comparison of the leading options as of 2025.

How to Choose

Choose Langfuse if you want an open-source, self-hostable solution with strong prompt management, you use multiple LLM frameworks, or you want to avoid per-trace pricing at scale. It is the most framework-agnostic option.

Choose LangSmith if your entire stack is LangChain-based and you need the tightest debugging experience for complex agent graphs. The automatic instrumentation for LangGraph is unmatched.

Choose W&B Weave if your team already uses Weights & Biases for ML experiment tracking and wants a unified platform. Avoid if you only do inference and don't need the full ML lifecycle tooling.

Choose Arize Phoenix if you need embedding drift detection and RAG-specific evaluation metrics, or if you are running a high-volume production system and want the open-source version to control costs.

Build a custom OTel stack if you have existing Grafana/Prometheus infrastructure, a platform team that can maintain it, and need to integrate LLM traces alongside non-LLM service traces in a single pane of glass.

4. Setting Up Langfuse in 5 Steps (Python)

Langfuse is the fastest way to get production-grade LLM observability running for a Python application. Here is the full setup from zero to traces in the dashboard.

pip install langfuse openai

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://cloud.langfuse.com  # or your self-hosted URL
OPENAI_API_KEY=sk-...

from langfuse.openai import openai  # drop-in replacement
from langfuse.decorators import observe, langfuse_context
import os

# All OpenAI calls are now automatically traced
# No other configuration needed — keys read from env

@observe()
def answer_question(question: str) -> str:
    """Traced function — every call creates a Langfuse trace."""
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": question}
        ],
        temperature=0.3,
    )
    return response.choices[0].message.content

@observe()
def handle_user_query(user_id: str, session_id: str, query: str) -> str:
    # Attach user and session context to the trace
    langfuse_context.update_current_trace(
        user_id=user_id,
        session_id=session_id,
        tags=["production", "v2-prompt"],
        metadata={"feature_flag": "rag-enabled"},
    )
    answer = answer_question(query)
    # Optionally score the trace immediately
    langfuse_context.score_current_trace(
        name="user-feedback",
        value=1,  # 1 = thumbs up, 0 = thumbs down
        comment="Automated pre-flight check passed",
    )
    return answer

from langfuse import Langfuse

langfuse = Langfuse()

if __name__ == "__main__":
    result = handle_user_query(
        user_id="user-42",
        session_id="session-abc123",
        query="What is the capital of France?",
    )
    print(result)
    langfuse.flush()  # Ensure all traces are sent before exit

5. Setting Up LangSmith with LangChain (Python)

LangSmith offers the deepest integration with LangChain and LangGraph, making it the natural choice for teams already invested in that ecosystem. Setup is almost entirely automatic once environment variables are set.

pip install langchain langchain-openai langsmith

LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
LANGCHAIN_API_KEY=ls__...
LANGCHAIN_PROJECT=my-production-app   # Optional: separate projects per environment
OPENAI_API_KEY=sk-...

With the environment variables set, every LangChain chain, agent, and tool invocation is automatically traced — no code changes required. Here is a more complete example that traces a ReAct agent with custom tool use:

from langchain_openai import ChatOpenAI
from langchain.agents import create_react_agent, AgentExecutor
from langchain_core.tools import tool
from langchain import hub
from langsmith import traceable

# @traceable decorator adds custom functions to the LangSmith trace tree
@traceable(name="fetch-stock-price", run_type="tool")
def fetch_stock_price(ticker: str) -> str:
    # Simulated — replace with real API call
    prices = {"AAPL": "$189.30", "MSFT": "$415.20", "NVDA": "$875.40"}
    return prices.get(ticker.upper(), "Ticker not found")

@tool
def stock_price_tool(ticker: str) -> str:
    """Returns the current stock price for a given ticker symbol."""
    return fetch_stock_price(ticker)

llm = ChatOpenAI(model="gpt-4o", temperature=0)
prompt = hub.pull("hwchase17/react")
tools = [stock_price_tool]

agent = create_react_agent(llm, tools, prompt)
agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=False,  # LangSmith handles the verbose logging
    return_intermediate_steps=True,
)

# This single invocation generates a full trace tree in LangSmith:
# Agent → LLM Call → Tool Call → LLM Call → Final Answer
result = agent_executor.invoke({
    "input": "What is the current price of NVDA stock?"
})
print(result["output"])

Adding Manual Feedback to LangSmith Traces

from langsmith import Client

client = Client()

# Retrieve the run ID from the trace (available via run metadata or callbacks)
client.create_feedback(
    run_id="<run-id-from-trace>",
    key="correctness",
    score=1.0,
    comment="Answer was accurate and cited sources correctly",
)

# Run a dataset evaluation against a benchmark
from langsmith.evaluation import evaluate

results = evaluate(
    agent_executor.invoke,
    data="my-golden-dataset",
    evaluators=["correctness", "conciseness"],
    experiment_prefix="gpt-4o-react-v2",
)

6. Building a Custom Dashboard with Grafana + Prometheus

For teams with existing Grafana infrastructure — or those who want maximum control over their observability stack — building a custom LLM monitoring pipeline with Prometheus and Grafana is a powerful option. This approach uses the OpenTelemetry Collector as the central pipeline, the opentelemetry-instrumentation-openai library for automatic instrumentation, and Prometheus as the metrics backend.

Architecture Overview

The data flow is: Python App + OTel SDK → OTel Collector → Prometheus (metrics) + Grafana Tempo (traces). The Collector scrapes metrics from your app, batches and exports them to Prometheus, and forwards trace spans to Tempo. Grafana reads from both and lets you correlate a latency spike in a Prometheus panel directly with the trace that caused it.

Step 1: Instrument your Python app with OpenTelemetry

pip install opentelemetry-sdk opentelemetry-exporter-otlp \
    opentelemetry-instrumentation-openai prometheus-client

from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

# Configure trace exporter → OTel Collector → Grafana Tempo
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://localhost:4317")
    )
)
trace.set_tracer_provider(tracer_provider)

# Configure metrics exporter → OTel Collector → Prometheus
metric_reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="http://localhost:4317"),
    export_interval_millis=15000,
)
meter_provider = MeterProvider(metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)

# Auto-instrument all OpenAI calls — adds gen_ai.* span attributes
OpenAIInstrumentor().instrument()

# Define custom LLM metrics
meter = metrics.get_meter("llm-app")
llm_cost_counter = meter.create_counter(
    name="llm.cost.usd",
    description="Total LLM cost in USD",
    unit="USD",
)
llm_latency_histogram = meter.create_histogram(
    name="llm.request.duration",
    description="LLM request latency in milliseconds",
    unit="ms",
)

Step 2: OTel Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
  resource:
    attributes:
      - key: service.name
        value: llm-production-app
        action: insert

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write
  otlp/tempo:
    endpoint: http://tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheusremotewrite]

Step 3: Grafana dashboard queries

Once data is flowing into Prometheus, create a Grafana dashboard with these key PromQL queries:

# P95 LLM latency (ms) over 5-minute windows
histogram_quantile(0.95, rate(llm_request_duration_bucket[5m]))

# Token usage rate (tokens/minute, split by model)
rate(gen_ai_usage_input_tokens_total[1m]) by (gen_ai_request_model)

# LLM API error rate percentage
100 * rate(gen_ai_errors_total[5m]) /
  rate(gen_ai_requests_total[5m])

# Estimated cost per hour (using OpenAI GPT-4o pricing)
(rate(gen_ai_usage_input_tokens_total{gen_ai_request_model="gpt-4o"}[1h]) * 0.0000025)
+ (rate(gen_ai_usage_output_tokens_total{gen_ai_request_model="gpt-4o"}[1h]) * 0.00001)

7. Cost of Observability Tools at Scale

Observability tools can themselves become a significant line item, particularly at high trace volumes. Here is a realistic cost breakdown at three scales of LLM application.

The Hidden Cost: Engineering Time

The true cost of a custom observability stack is not infrastructure — it is the platform engineering time to build and maintain it. Budget 2–4 weeks to build an initial OTel pipeline and Grafana dashboard, plus ongoing maintenance of roughly 4–8 hours/month. For teams without a dedicated platform engineer, Langfuse Cloud or LangSmith typically pays for itself in saved engineering hours until trace volumes exceed 1M/month.

Cost Optimization Strategies

The most effective cost lever is trace sampling. You rarely need 100% of production traces for dashboards. A 10% sample gives statistically valid latency histograms and error rates. Always capture 100% of errors, 100% of high-latency outliers (p99+), and a configurable sample of successful traces. Combine head-based sampling (probabilistic, cheap) with tail-based sampling (keep traces that meet specific criteria after completion) for the best coverage-to-cost ratio.

Frequently Asked Questions