Monitoring AI Systems in Production: Key Metrics

Traditional software either works or it doesn’t. A broken checkout page throws an error, a 500 shows up in your logs, someone gets paged. AI systems don’t fail that cleanly. Accuracy drifts a few percent a week, response quality slips, latency creeps up: and by the time a customer complains, the problem’s been running for a month. Monitoring AI systems properly means watching for decline, not just downtime.

This matters more than it did even a year ago. UK SMBs are shipping AI-powered chatbots, invoice extraction tools, and support triage agents into production without the observability habits that traditional software teams take for granted. If you’re running any of these, here’s what to actually track, how to set sane alert thresholds, and where off-the-shelf tools fit.

Why AI Monitoring Differs From Traditional Software

Traditional uptime monitoring is binary, the server responds or it doesn’t. AI systems introduce three problems that binary monitoring can’t catch.

No pass/fail line. A large language model can return a technically valid, well-formatted response that’s still wrong, outdated, or subtly off-brand. Nothing throws an error. Your dashboard stays green while quality quietly degrades.

Stochastic outputs. The same input can produce different outputs on different runs. That’s expected behaviour for most models, not a bug, but it means you can’t monitor for “did it return the same answer as yesterday.” You need distribution-level thinking, not single-response checks.

Model drift and input shift. The world changes faster than most models get retrained. New product lines, new customer language, seasonal patterns: all of it shifts the inputs your model sees, and its outputs degrade in response. According to industry coverage of production AI monitoring, a mature monitoring stack tracks input distributions, output distributions, and downstream business metrics as three separate layers, because a problem in one doesn’t always show up in the others straight away.

None of this means AI systems are unmonitorable. It means the metrics and alert logic have to be different from what you’d set up for a normal web app.

The Five Metrics Every AI System Needs

Start here regardless of what the system does: a chatbot, an AI agent doing invoice extraction, or a recommendation engine.

1. Accuracy or quality score

Some measurable proxy for “is this output good.” For structured tasks (extraction, classification), this can be a straightforward accuracy percentage against known-correct data. For open-ended generation, you’ll need a scoring approach: human spot-checks, a secondary model grading outputs against a rubric, or user feedback signals (thumbs up/down, edit rate).

2. Latency: p50, p95, and p99

Average latency hides the problem. If your median response time is 800ms but your 99th percentile is 12 seconds, 1 in 100 users is having a genuinely bad experience, and that’s the number that drives complaints. Track p50 (typical), p95 (most users’ worst case), and p99 (your outliers) separately.

3. Throughput

How many requests or operations the system handles per minute/hour, and whether that’s trending with demand or against capacity. A drop in throughput with steady demand usually means something upstream, an API rate limit, a queue backing up, not a model problem.

4. Error rate

Not just hard failures (timeouts, API errors) but soft failures too, empty responses, responses that fail a format check, requests that needed a fallback path. Group these separately; a spike in hard errors and a spike in soft failures point to different root causes.

5. Cost per operation

AI systems have a cost profile traditional software doesn’t, every request has a token or compute cost attached. Track cost per operation alongside volume so a traffic spike doesn’t become a budget surprise. This is also your earliest warning sign of inefficient prompts or unnecessary retries.

Quick win: if you’re monitoring nothing right now, start with error rate, p95 latency, and cost per operation this week. All three are cheap to instrument and catch the most common production issues.

Detecting Model Drift

Model drift is the gradual mismatch between what a model was built to handle and what it’s actually seeing in production. It’s the AI-specific failure mode traditional software monitoring has no equivalent for, we cover the causes and mitigations in full in model drift explained.

What causes it:

• Your customer base or product mix changes, shifting the language and context inputs contain
• Seasonal patterns the model never saw during training or tuning
• Upstream data pipeline changes that alter formatting, units, or field structure
• The underlying model provider updates their model without warning (common with hosted LLM APIs)

How to detect it:

1. Reference dataset comparison. Keep a fixed, representative sample of “known good” inputs and expected outputs. Run it against your live system on a schedule (weekly is reasonable for most SMB use cases) and track score degradation over time.
2. Output distribution monitoring. Watch for shifts in response length, sentiment, category distribution, or confidence scores. A sudden change in the shape of your outputs, even without an obvious error, is an early drift signal.
3. Input distribution monitoring. Track what’s actually coming into the system. If your support chatbot starts getting a wave of questions about a product it’s never seen, that’s a leading indicator, not a lagging one.

Drift is rarely sudden. Left unchecked, it’s the difference between an AI system that quietly underperforms for two months before anyone notices, and one where you catch the trend in week one and retrain, re-prompt, or adjust before it costs you customers.

Alerting Strategy for AI Systems

The instinct with new monitoring is to alert on everything. With AI systems, that guarantees alert fatigue within a week, because some variance is normal, expected behaviour, not a fault.

Threshold-based vs anomaly-based:

• Use threshold alerts for hard limits that should never be crossed, error rate above X%, p99 latency above Y seconds, cost per operation above Z. These are appropriate for a page or immediate notification.
• Use anomaly-based alerts for metrics with natural variance, quality scores, output distribution shifts, drift indicators. These belong in a daily or weekly digest, not a page, because a single anomalous reading is often noise.

What warrants a page vs a report:

The goal is a small number of alerts that always matter, not a large number you learn to ignore. If your team starts muting notifications, your thresholds are wrong, recalibrate rather than add more noise on top.

Tools and Approaches

You don’t need to build an observability platform from scratch. A handful of purpose-built tools cover most SMB use cases:

• Arize AI / Arize Phoenix: dashboards for latency percentiles, token usage, cost per request, and embedding drift monitoring, with anomaly detection built in.
• Helicone: lightweight LLM request logging and cost tracking, a reasonable starting point if you’re only using one or two model providers.
• Custom dashboards: for simpler systems (a single extraction pipeline, one chatbot flow), a basic dashboard pulling from your own logs is often enough, and avoids paying for observability tooling you’ll only use 10% of.

Note that WhyLabs, previously a common recommendation in this space, was acquired by Apple in early 2025 and its commercial product was discontinued, if you’re evaluating tools now, that’s one to skip.

What to look for in any AI observability tool:

• Native support for percentile-based latency (not just averages)
• Cost tracking per request/operation, not just aggregate spend
• Some form of drift or distribution monitoring, even basic
• Alerting that lets you separate “page me now” from “tell me this week”
• Reasonable setup time, if instrumenting the tool takes longer than building the feature it’s monitoring, it’s the wrong tool for an SMB-scale system

For context on scale: companies like Netflix and Uber run dedicated ML platform teams with custom-built monitoring infrastructure. Most SMBs don’t need that, a well-configured off-the-shelf tool, checked against the five metrics above, covers 90% of what actually causes production incidents at this scale.

What This Looks Like in Practice

A mid-sized UK recruitment firm running an AI-powered CV screening tool caught a drift issue early because they’d set up a weekly reference-dataset check. Screening accuracy on the reference set dropped 8 points over three weeks, no errors, no downtime, nothing that would have shown up in standard uptime monitoring. The cause was a new CV template their applicant tracking system had started using, which shifted input formatting just enough to confuse the extraction step. A prompt adjustment fixed it in an afternoon. Without the reference check, they’d likely have noticed only when a hiring manager flagged bad shortlists, weeks later, after real candidates had already been mis-screened.

That’s the pattern worth internalising: the monitoring doesn’t need to be sophisticated. It needs to exist, and it needs to run on a schedule, not “whenever someone remembers.”

Getting Monitoring in Place

If you’re running AI systems in production without structured monitoring, start small rather than trying to instrument everything at once:

1. This week: add error rate, p95 latency, and cost-per-operation tracking
2. This month: build a reference dataset and run it weekly against live output
3. This quarter: set threshold alerts for hard failures and a weekly digest for drift and quality trends

Fernside Studio builds and supports AI systems for SMB teams, and monitoring is part of how we keep them reliable after launch rather than something bolted on later. Our managed systems support covers ongoing oversight through ticketed support: no retainers, just monitoring and fixes billed against the work actually done.

If you’re running an AI system that’s flying without instrumentation, or you inherited one that’s showing signs of drift, get in touch and we’ll help you work out what to monitor first.

Related reading

• Model drift explained: why accuracy declines after launch, and how to catch it early
• Testing AI outputs, building golden sets and regression checks that make monitoring meaningful
• When AI gets it wrong, handling the failures monitoring surfaces

Sources

• AI Monitoring in Production 2026: LLM Observability & Drift Detection: ValueStream AI, 2026
• LLM Observability for AI Agents and Applications: Arize AI, 2026
• AI Model Monitoring in Production: Drift and Decay in 2026, Logiciel, 2026
• What is AI Model Drift? Best Practices & How It Works, Logz.io