AI

LLMOps is where enterprise AI quietly breaks after the demo

November 17, 20267 min read
On-call engineer at a desk in a planted office late in the day, for a post on LLMOps after the generative AI demo goes live

LLMOps is the discipline that keeps an LLM application working after launch: pinned models, an evaluation set, versioned prompts, monitoring, cost tracking and a named owner. Demos often ship without it, and the gaps show by month three. Future Works builds it in from go-live, since a portion of our fee is staked on value Finance verifies by week 12.

On October 14, 2026, OpenAI notified developers that GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini will be removed from its API on January 14, 2027, and that fine-tuned versions built on them follow the same schedule, three months of notice in all. Any team that demoed a generative AI assistant on one of those models this year now has a forced migration on its calendar, whether or not anyone planned for one. A team with LLMOps in place will run its test set against the replacement model and read the results, and a team without it tends to hear about the change from users, after the first odd answer reaches a customer or a manager.

What is LLMOps, and how is it different from MLOps?

MLOps grew up around models you train yourself. You own the weights, you decide when to retrain, and the thing that drifts is mostly the data coming in. We covered the classic controls for that world before, and most of them still apply.

Large language models change four things. You usually rent the model through an API, so the provider decides when it changes and when it retires. The behavior lives in prompts and in whatever context you retrieve and feed in, which means a one-line edit to a system prompt is a release, even though nobody treats it like one. The output is open text, so "accuracy" stops being a single number and becomes a judgment against examples of what a good answer looks like. And the cost is metered per token, so it grows with usage and with every extra page of context you stuff into the prompt. LLMOps is the set of habits that handles those four things. None of it is exotic, and it tends to get skipped because a demo can look excellent without any of it.

What breaks in the first month after a generative AI demo goes live?

The first month usually feels fine. A small group of friendly users tries the assistant, the volume is low, and the feedback comes in over chat. The demo was judged by eye in a meeting, so there is no written set of test cases, and nobody has defined what a wrong answer is in terms the business would sign.

The damage in month one comes from prompt editing. A user complains about one answer, someone close to the build tweaks the system prompt to fix it, and two other cases nobody is checking get worse. With no evaluation set, there is no way to see that trade happen. The prompt in production slowly drifts away from the one that won the demo, and there is no record of who changed what.

What happens when an LLM provider changes or retires a model?

Model providers update and retire models on their own schedule. One well-documented example of what that does to an application is a 2023 study by Lingjiao Chen, Matei Zaharia and James Zou at Stanford and UC Berkeley. They tested the March 2023 and June 2023 versions of GPT-4 on the same tasks. On telling prime from composite numbers, accuracy fell from 84% to 51% between the two versions, and the authors found that GPT-4's ability to follow user instructions had decreased, one common factor behind many of the shifts they saw. Their conclusion was that the "same" LLM service can change a lot in a short time, and that it needs continuous monitoring.

Pinning your application to a dated model snapshot, instead of a moving alias, helps. It only buys time, because snapshots retire too (the January 2027 removals include dated snapshots). In month two, the team without an evaluation set gets a vague complaint ("it's worse this week") and has no way to tell whether the cause is the model, the prompt someone edited, or a change in the documents being retrieved. They argue about it in a meeting, and the users drift back to the spreadsheet.

Why do LLM costs rise after launch, and who owns the fix?

By month three, usage has grown, and so has the context in each request, because every fix seems to add another paragraph of instructions or another retrieved document. Per-token pricing turns both into cost. Gartner predicted in July 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, and escalating costs were one of the four reasons it named, alongside poor data quality, inadequate risk controls and unclear business value. Rita Sallam, the Gartner analyst presenting it, said that "costs aren't as predictable as other technologies." Cost per token on its own is a misleading number, as I wrote when modeling cost against latency and quality. For LLMOps, that means tracking the fully loaded cost of each completed task (API spend plus the human review and rework around it) from day one.

Month three is often when the first serious failure arrives and someone asks who owns the assistant. The data science team that built it reports to one leader, the operations team that uses it reports to another, and the vendor that hosted the demo has moved on. I made the same point about stalled pilots: when nobody owns the call, nobody makes it. An LLM application in production needs a named person with the authority to roll back a prompt or a model version without convening anyone.

What does LLMOps need before go-live?

The controls from the MLOps world carry over (testing on every change, versioning, monitoring, an audit trail), adapted for a model you rent and steer with prompts. Before a generative AI application touches a real workflow, five things should exist.

  1. An evaluation set: real cases pulled from the workflow, each with what a good answer looks like, scored automatically on every prompt, model or retrieval change.
  2. Version pinning and a migration plan: call a dated model snapshot, write down its retirement date, and rehearse the move to its replacement against the evaluation set before you are forced to.
  3. Prompts and configuration under version control, with review, so a prompt edit is a release with a name and a date on it.
  4. Production monitoring: quality scores on sampled live traffic, latency, refusals and errors, plus a regular human review of a small sample.
  5. A named owner with on-call duty and the authority to roll back, and a fully loaded cost-per-task number that person reviews every week.

How does Future Works run LLMOps?

At Future Works, controls like these are part of AI operationalization, our term for moving AI out of the pilot and into the parts of the business that change the P&L. They sit inside our proprietary AI-native operating system: the layer of AI agents, senior operators, instrumentation and guardrails that runs the work in production, with every request, decision and result on the ledger. The senior experts who direct those agents are listed by name in the SOW, and the client signs off before any of them changes.

The timing matters too. The client's Finance team signs an Approved Value Baseline in week 2 of our 12-week cycle, the work goes live from week 4, and Finance verifies the value by week 12. That puts the first two months of live running, months one and two in the story above, inside the cycle, so the LLMOps controls have to work from the day the work goes live, or the number Finance checks will not hold. Because a portion of our fee is staked on a value target the client's Finance team validates, a prompt that drifts in month one costs us as well. Our first cycle puts a working workflow into the client's live operation on real volume. If you are weighing this against an integrator's managed service, our comparison page puts the two side by side.

Keep reading

Related Articles

What is a digital maturity assessment?

AI

Generative AI consulting sells slides. The P&L needs shipped software.

AI

How to Calculate ROI for a Digital Twin (Factory Model)

AI