Building an impressive AI demo has never been easier. Connect to a large language model, write a good prompt, add a bit of UI, and within a week you have something that gets stakeholders excited. But then it goes live…

The model provider's invoice is higher than anyone expected. A user reports a strange answer that nobody can reproduce. The model you built on gets deprecated. Someone tweaks a prompt to fix one case and quietly breaks five others.

That is the gap between a demo and a product, and closing it is what LLMOps is about: the practices and tooling to run LLM-powered applications reliably, affordably and responsibly over time. Here is how we approach it at Hatch.

Why LLMs need their own "Ops"

LLM applications behave differently from traditional software:

• The output isn't deterministic. The same input can give different answers, so classic tests only get you so far.

• Every request costs money. You pay per token, and small changes in prompts or usage can have a big effect on the bill.

• Prompts are code, but often without versioning, review or testing.

• The landscape moves fast. New models arrive every few months and older ones are retired.

• Quality is hard to measure. "Is this answer good?" is a much harder question than "did the endpoint return a 200?"

When does it make sense?

We don't apply all of this to every project. If an application uses an LLM for one small feature, such as summarizing a text or suggesting a title, a full LLMOps setup would be overkill. Solid engineering practices and some basic monitoring are usually enough.

LLMOps is for AI-centric applications, where LLMs drive the core of the product: assistants, agents, document processing pipelines, or anything where the quality of the AI output determines whether the product works. That's where the complexity, the costs and the risks add up, and where LLMOps pays for itself many times over. For those applications, we make it part of the project from the start, not something to add later.

Our approach in six practices

1. We never tie ourselves to one model provider

Our applications don't call model providers directly. Every request goes through an abstraction layer, usually LangChain or LiteLLM depending on the project, which gives us one consistent interface to OpenAI, Anthropic, Mistral, Google or an Azure deployment.
This lets us switch to a newer, cheaper or better model by changing configuration instead of code. We can also set up fallbacks for when a provider has an outage or hits a rate limit, and keep API keys and access in one place.

2. We trace everything from day one

Every request is logged as a trace: the prompt, the retrieved context, tool calls, the model's answer, latency and cost. For multi-step chains and agents, we see each step separately.

When a user reports a bad answer, we don't guess. We open the trace, typically in Langfuse, and see exactly what happened. Both LangChain and LiteLLM integrate with it out of the box, so tracing costs the development team almost no extra effort. Starting on day one matters: the first weeks after launch are when you learn the most, and you don't want to lose that data.

3. Prompts are versioned and managed

Prompts change more often than any other part of an LLM application, and they have the biggest effect on its behaviour. So we manage them like we manage code: versioned, with clear ownership, and with the option to roll back.
How we do that depends on the project. Sometimes prompts live in a prompt management
platform, so they can be changed without a new deployment. Other times we keep them as Jinja templates ( .j2 files) in the codebase, versioned in Git and reviewed like any other code change.

For some projects we've built a custom versioning system that fits the client's workflow.
Whichever we choose, we always know which prompt version is running where, and we can link every trace back to it.

4. We evaluate before we ship

This is where most teams can gain the most. For each use case, we build a dataset of
representative examples, often taken from real usage. Every change to a prompt or model is scored against that dataset before it reaches users. Depending on the use case, scoring is automated (including LLM-as-a-judge), rule-based, or done by humans.
As a result, we decide on evidence rather than gut feeling. When a new model comes out, we can tell within hours whether it's actually better for this use case.

A test bench for our clients. For some projects we go one step further and build a "test bench": A simple interface where the client's own domain experts can adjust prompts, compare models side by side, and run them against the evaluation dataset. Nobody knows what a good answer looks like better than the people who use the system every day. A test bench lets them keep quality up and try improvements without having to wait for a developer.

5. We make costs visible

We track cost and usage per project, per feature and, where relevant, per client or user. Knowing what each part of the application costs prevents surprises and supports better design decisions.
It often shows that a smaller, cheaper model handles a task just as well, or that caching would pay off.

When a project needs it, we go further and set hard limits. With LiteLLM, for example, budgets and rate limits can cap costs per project, team or user before they get out of hand.

6. We close the loop with feedback

Real usage is the best source of improvement. We collect user feedback (a thumbs up or down, corrections, escalations) and link it to the trace it belongs to. Bad answers become new test cases in our evaluation datasets, so the same mistake doesn't come back.

This turns an AI feature from a one-off build into a product that keeps getting better.

Why we favour open-source tools

You'll notice that the tools we reach for most, LangChain, LiteLLM and Langfuse, are all open source. That's a deliberate choice, even though we don't always self-host them:

• No lock-in at any layer. It would make little sense to avoid lock-in to a model provider and then get locked into an operations platform.

• Transparency. We can see how the tools work and trust what they report.

• Options for our clients. If a client needs to run everything in their own infrastructure later, for example for compliance reasons, open source makes that possible without starting over.

• Strong communities. These projects move fast and keep up with a fast-changing ecosystem.

That said, the tools serve the practices, not the other way around. If a client already has a
platform in place, we'll happily work with it.

Conclusion

The value of an AI feature isn't in the demo. It's in running it reliably, safely and cost-effectively for months and years. Model flexibility, tracing, prompt versioning, evaluations, cost visibility and feedback loops are how we get there at Hatch, whichever tools a project ends up using.

Building something with LLMs, or struggling to get a prototype into production? Let's talk.