Shipping LLMs to Production: Lessons from the Trenches

Prototyping with LLMs takes an afternoon. Shipping them to production, where real users depend on them, is a different discipline entirely. Here's what I've learned leading teams that run AI features in production.

The demo-to-production gap

An LLM demo that works 9 times out of 10 feels magical. In production, a 10% failure rate means thousands of broken user experiences per day. The gap between "works in the demo" and "works at scale" is where most AI projects die.

Three things close that gap:

1. Treat prompts like code

Prompts are logic. They belong in version control, they need code review, and every change needs regression testing.

• Keep prompts in the repo, not in a dashboard someone edits by hand.
• Build a golden dataset: 50–200 real examples with expected outputs.
• Run evals on every prompt change, like unit tests. A one-word edit can silently degrade a distant use case.

2. Design for failure, because failure is guaranteed

LLM calls fail in ways traditional APIs don't: timeouts, rate limits, malformed output, hallucinated fields, refusals.

• Validate structure: force JSON output with a schema, and validate it. Retry with the validation error in the prompt.
• Set timeouts aggressively and always have a fallback path: a simpler model, a cached answer, or a graceful "not available" state.
• Never let the model write directly to anything important. Human-in-the-loop or constrained tool use for anything with side effects.

3. Measure quality, not just uptime

Your observability stack needs new dimensions:

| Traditional metric | LLM-era addition | |---|---| | Latency, error rate | Token cost per request | | Throughput | Eval scores over time | | Availability | Output validation failure rate | | n/a | User feedback signal (thumbs, retries, abandons) |

If you can't answer "did last week's model upgrade make answers better or worse?", you're flying blind.

The organizational lesson

The biggest mistake I see is treating AI features as a side project of the data team. LLM features are product engineering: they need the same ownership, on-call, and quality bar as your payments flow. The teams that get this right ship AI that users actually trust.

Start small, instrument everything, and earn your way to more autonomy for the model. Reliability is the feature.