Token Usage, Latency, and Cost Tracking
On every model span log input, cached, reasoning, and output tokens, plus TTFT and total latency. Cost is dollars per successful task — including retries and loops — not the sticker price per thousand tokens.
Explain like I'm new to AI
Tokens are the fuel gauge. Latency is the stopwatch. Dollars are what finance sees. If you only log "the call took 2s," you cannot tell a cache miss from a reasoning-model think from an agent that looped four times.
~$0.001 this request
Stable tools + instructions hit the prompt cache. Input still '800' on the span — billable uncached is 80. Watch cache-busting edits.
Input tokens: what you sent. Cached input: prefix that hit a prompt cache (you still record the full input size; the bill is the uncached remainder). Reasoning / hidden tokens: 2026 thinking models spend tokens you never show; they dominate the invoice if you don't log them. Output tokens: the visible (and tool-call) text.
TTFT (time to first token) is chat UX. Total time is when the JSON is valid or the tool loop finished. Alert on p95, not the mean.
Mental model
A taxi meter that charges waiting, highway, and tolls separately. If the receipt only says "trip," you cannot tell why Tuesday was 10× Monday.
How it works
- Attributes on the LLM span:
input_tokens,output_tokens,cached_input_tokens,reasoning_tokens,model,ttft_ms,duration_ms. - Cost = provider formula(model, those counts) + tools + retrieval + your own eval judges if you score online. Roll up to $ / successful task (the previous category's knob).
- Dimensions:
prompt_version,route(easy vs hard),tenant. Budgets and alerts per dimension. - Alerts that matter: agent loop count spike, cache hit rate collapse (someone edited the prefix), reasoning tokens per request exploding, output length runaway.
- Streaming: TTFT at first byte; don't mark success until the stream completed or JSON parsed.
Real-world example
Invoice +40% week over week. Metrics show input tokens flat. New field: reasoning_tokens after a model swap. You route FAQs off the thinking model. Eval quality on FAQs holds. The waterfall in the last visual's "reasoning" preset is that incident.
Technical explanation
Provider usage objects differ; normalize into those four token fields so dashboards survive a vendor change. Cached tokens require you to know whether the prefix actually matched (see Context Caching).
Retries and fallbacks: a failed expensive call still costs. Count it on the parent span.
Reliability vs cost: timeouts you retry are a cost feature. Track retry_count on the trace.
Don't use $ / 1k as the product KPI. A cheap model that loops is expensive (quality–latency–cost lesson). Observability is how you see the loops.
Common mistakes
Tracking only output tokens. Reasoning and uncached input are where 2026 bills hide.
- Mean latency as the SLO (hides p95).
- No cache-hit metric — prefix edits silently 5× prefills.
When to use it
- Every production model call, from day one. Especially agents and reasoning models.
When NOT to use it
- Don't "save money" by dropping token fields from traces. You'll fly blind into the invoice.
- Don't optimize cache hit rate if it means stuffing unused tools and busting quality (eval still gates).
Alternatives
- Provider invoices are a lagging monthly lump. They are not request-level tracking.
Quick quiz
Related concepts
- Quality, Latency, Cost, and Reliability — Production AI is a tradeoff of quality, latency, cost, and reliability. Optimize dollars per successful task under an SLO — not always the smartest model.
- Context Caching — Prompt caching reuses the KV cache of a stable prefix so you don't re-prefill tools and instructions every turn — any change in that prefix busts the cache.
- Secrets Management and Sandboxing — Keep keys out of prompts, run untrusted work in a jail with an egress allow-list, and cap loops and spend so agents cannot become a denial-of-wallet.
- Caching and Cost Optimization — Three inference caches — exact replay, prefix KV, semantic — plus $ per successful task. Key on prompt version; similar is not the same answer.
- Latency vs Throughput — Latency is one user's wait (TTFT, ITL, queue); throughput is tokens per second per GPU. Bigger batches raise throughput and hurt latency — pick the SLO first.
Last reviewed: 2026-09-04 · Written by ByHeart AI · Reviewed by ByHeart AI