You're Flying Blind on LLM Costs (And It's Expensive)

The OpenAI dashboard shows you a number. Let's call it $47,320. That number went up 38% this month, which is roughly the rate your revenue didn't grow. Your CFO asks what it was spent on. You have no idea.

This is where most AI-first companies sit in 2026. Not because engineers are lazy — because the observability stack for LLM spend does not yet exist by default. Every cloud has per-service cost attribution; every database has per-query cost attribution; the LLM layer has "here is your invoice for the entire month, good luck."

The gap is not cosmetic. It turns out the CFO-facing question ("what was spent on") maps to the engineer-facing question ("which features are expensive") maps to the business question ("are our unit economics fixable or broken?"). Companies flying blind for a year end up with unit economics in tatters and no blueprint for repair.

The three questions you can't answer today

1. What does each feature cost per user?

You ship a chatbot, an AI-search feature, and an agent-based onboarding flow. Your invoice is $47K. How much was each? If you're using the OpenAI SDK directly, the honest answer is "no idea" unless you set up per-feature API keys, per-feature wrappers, and a manual reconciliation pipeline. Most teams don't bother because it's a week of engineering time.

2. Which task types dominate the bill?

Across all those features, how much went to heavy reasoning vs. lightweight extraction? The bill doesn't tell you that; it tells you per-model tokens. Every team we've onboarded has had a moment where they see their cost-per-task-type chart for the first time and go "oh, 62% of our spend was on simple classification tasks that could have run on Haiku." That's the routing opportunity staring them in the face.

3. Did the quality get better or worse when we made that change?

Last week you switched a workflow from Sonnet to Haiku. Bill went down 8%. Did quality drop? Did users notice? Did downstream metrics — support ticket creation, onboarding completion, feature retention — shift? Without a quality signal pipeline, you have no idea until a customer complaint shows up, by which point the damage is two weeks old.

The minimum viable observability stack

Here's the stack most teams end up with after a year of scars. You can build this yourself. We obviously also build it for you as part of the product.

Layer 1: Per-request trace logging

Every LLM call logs: {request_id, user_id, feature_id, task_type, model, input_tokens, output_tokens, latency_ms, cost_usd, timestamp}. Send to a columnar store (BigQuery, ClickHouse, Snowflake). This is the foundation. Without it you have nothing.

The most common mistake: logging just user_id and cost. You need feature_id and task_type to answer the interesting questions. Task type requires a classifier; feature ID requires instrumentation discipline in the app layer.

Layer 2: Aggregated dashboards

On top of the trace store: a dashboard that slices spend by feature, by task type, by model, by user cohort, and by day. This is a week of chart-building if you have a BI stack in place, a month if you don't. The three charts that actually matter:

• Cost per feature over time. Is it trending up more than usage? That means cost-per-user is growing — your unit economics are deteriorating.
• Cost per task type. Is 60%+ of spend in "easy" categories? That's the routing win. Also: is there a category growing faster than revenue?
• Cost per user cohort. Does your $10/mo tier user generate $3 or $12 in API cost? (We've seen both. The $12 one was a company bleeding money.)

Layer 3: Quality signals

For every request, collect downstream signals that approximate quality. The four that generalize:

• Response length. A strong weak signal — large unexpected drops in output length often correlate with worse outputs.
• Tool-call success rate. For agentic workloads, did the model call the right tools in the right order? Did the tools succeed?
• Explicit user feedback. Thumbs up/down in the UI, or implicit (did the user regenerate, retry, or abandon?).
• Downstream retry. Did the user's agent immediately call a frontier model after this one to "fix" the output? That's a strong regression signal.

Layer 4: Alerts

On top of the signals: alerts when the quality indicators shift after a routing change, model swap, or provider outage. If you're only looking at cost, you'll pat yourself on the back for a cost-cut that quietly tanked quality for two weeks.

Buy vs. build (the real math)

You can build all four layers. Back-of-envelope:

• Layer 1: 1–2 weeks of engineering. Straightforward.
• Layer 2: 2–4 weeks. The hard part is defining feature IDs and task types — that's a cross-functional decision, not a pure coding task.
• Layer 3: 6–12 weeks. The quality signal pipeline is where most in-house projects stall.
• Layer 4: 2–3 weeks, with ongoing tuning forever.

Call it a full engineering quarter of focused work, plus two quarters of ongoing tuning and bugfix. For a startup, that's a hire. For an enterprise platform team, that's a squad.

Or you point at api.kairosroute.com and get all four layers bundled. This is literally what KairosRoute is. The router is the wedge; the observability is the product.

What "good" looks like

A couple of screenshots' worth of verbal description so you know what to aim for, whether you build or buy.

The cost-per-task-type chart. Six bars, one per task category. You should be able to eyeball which categories are oversized for the value they deliver. If 35% of your spend is "summarization" and you suspect a lot of those summaries are going to waste, that's your first cost-cut conversation.

Per-feature cost trend. Eight lines, one per major feature. Features that grow cost faster than revenue have broken unit economics; features that are flat while usage grows are well-optimized. Call out the outliers weekly.

Per-agent attribution. If you run agents — even just a handful — show one chart per agent with cost-per-task, completion rate, and user satisfaction. The worst agents are usually 5–10x more expensive than the best agents doing similar work. Routing can help; architecture changes matter more.

Where this ends up

The companies that have this observability in place run AI at 60–80% gross margin. The ones flying blind run at 20–40%. That's the same product, same market, same revenue; the difference is visibility. You can't optimize what you can't see.

If you'd rather skip the DIY, sign up. The free tier gives you the dashboard. Point your OpenAI client at our base URL and you get Layer 1–4 immediately.