Fernside Studio
Background
Archive
Journal Entry

Testing AI Outputs: QA for Non-Deterministic Systems

Documented
Capacity
9 MIN READ
Domain
Website Performance & Ops

Run the same prompt through an AI system twice and you can get two different, both correct, answers. That breaks the basic assumption behind most software testing: same input, same output. So how do you actually know if your AI is working? Here’s the testing framework we use for evaluating AI outputs on client systems, built for non-deterministic behaviour rather than against it.

Why Traditional Testing Fails For AI

Unit tests assert one exact output. AI systems don’t behave that way, and forcing them into pass/fail testing produces false confidence or constant false alarms.

Four differences matter most:

  • Non-deterministic outputs. The same input can legitimately return varied phrasing, structure, or reasoning paths. A test expecting an exact string match will fail correct answers constantly.
  • Multiple valid answers. A customer support reply, a summary, or a generated proposal section can be “right” in several different forms. There’s rarely one canonical correct output to assert against.
  • Gradual degradation, not binary failure. Traditional software either works or throws an error. AI systems drift: accuracy slips a few percent a week as source data changes, a model is swapped, or a prompt is tweaked. Nothing crashes. Quality just quietly erodes. (We unpack this failure mode in model drift explained.)
  • Context dependency. Output quality depends on what’s fed in at that moment, retrieved documents, conversation history, user data. The same prompt logic can perform differently across contexts, so a single test rarely represents production reality.

This is why QA content built for traditional software (unit tests, integration tests, CI green ticks) doesn’t transfer cleanly to AI features. You need evaluation, not just testing.

Building Evaluation Datasets (“Golden Sets”)

The foundation of any credible AI QA process is a curated set of input/output pairs where a human has verified the correct or acceptable answer. Industry practice calls these golden datasets: a fixed benchmark you can run your system against repeatedly and compare results over time, as described in Statsig’s overview of golden dataset evaluation standards.

What goes into a golden set:

  1. Representative real cases: pull actual production inputs (enquiries, documents, questions) rather than inventing hypothetical ones.
  2. Edge cases: ambiguous requests, missing data, adversarial inputs, and anything that previously caused a wrong answer.
  3. A human-verified expected answer or acceptance criteria for each case, not always one exact string, but a rubric of what counts as correct.

How many do you need? Fewer than you’d think to start. 30 to 50 well-chosen cases covering your main scenarios and known failure modes gives you a usable baseline. Growing it over time matters more than starting big, every real production failure should get added as a new test case, so the set gets sharper the longer the system runs.

Maintenance is not optional. As noted in Techment’s research on golden datasets for GenAI testing, datasets need version control that maps to prompt and model versions, and should be continuously updated as content domains shift, otherwise you’re testing against a stale picture of what “correct” looks like.

A useful middle ground, per dev.to’s comparison of random sampling vs golden datasets, is to combine both: random sampling from live traffic surfaces new failure types quickly, while a lean golden set acts as the deterministic gate before anything ships.

Scoring Approaches: How To Actually Grade An Answer

Once you have test cases, you need a way to score outputs against them. Four common approaches, in order of rigour:

Binary (pass/fail) Simplest to implement. Works well for structured tasks with a clearly correct answer, did the system extract the right invoice total, did it classify the ticket into the right category. Poor fit for open-ended generation where “close enough” answers get unfairly failed.

Graded scoring (1 to 5) A human or automated grader scores each output on a scale against a rubric (accuracy, tone, completeness, relevance). Gives you nuance a binary test can’t, you can see a system sliding from consistent 5s to consistent 3s over a month, which is exactly the kind of gradual degradation binary tests miss entirely.

Semantic similarity Compares the meaning of the generated output to a reference answer using embeddings, rather than exact text matching. Useful for catching outputs that are correct in substance but phrased differently. Weak at catching factual errors dressed up in plausible-sounding language, similarity scoring rewards fluency, not truth.

LLM-as-judge Using a second AI model to grade the first model’s output against a rubric. Scales far better than human review and is now standard in evaluation tooling. But treat it carefully: recent research found frontier judge models fail more than half of advanced bias tests, and no judge model was reliable across all benchmark types, according to Adaline’s analysis of LLM-as-judge reliability and bias. A broader academic review reached a similar conclusion, reliability (consistent scoring) doesn’t guarantee validity (correct scoring), per this large-scale evaluation of LLM-as-a-judge models. Use LLM-as-judge for volume and speed, but validate it periodically against human judgement, don’t let it run unchecked as your only quality signal.

In practice: most working evaluation setups blend two of these, for example, binary checks on structured extraction fields plus graded LLM-as-judge scoring on the generated prose around them.

Regression Testing For AI Systems

The whole point of a golden set is to run it every time something changes, a new prompt version, an underlying model swap, a change to the data the system retrieves from. This is regression testing applied to AI, and it’s the single biggest gap we see in client systems that were built quickly and never revisited.

A practical regression cycle:

  1. Run the full golden set before any change ships.
  2. Run it again immediately after.
  3. Compare aggregate scores, not just individual cases, a 3% dip across the board is easy to miss case-by-case but obvious at the aggregate level.
  4. Flag any case that dropped from pass to fail for manual review before deployment.
  5. Store results with a timestamp and version tag so you can trace exactly when quality shifted and why.

This is what turns AI QA from a one-off launch check into an ongoing operational habit: the same discipline good teams apply to page speed monitoring or uptime, just applied to output quality instead of infrastructure.

Human-In-The-Loop Evaluation

Automated scoring should never be the only signal. Sampling a percentage of real production outputs for human review, weekly for high-stakes systems, monthly for lower-risk ones, catches what automated scoring misses: the answer that’s technically accurate but tone-deaf, the edge case your golden set never anticipated, the slow creep in phrasing that a customer would notice before a dashboard would.

Build this into a simple loop:

  • Sample a fixed number of production outputs each week
  • A human rates them against the same rubric used for the golden set
  • Anything scored poorly gets added to the golden set as a new test case
  • Repeat

This closes the gap between “our tests pass” and “our customers are getting good answers”, two things that drift apart quietly if nobody’s watching.

Common Mistakes We See

  • Testing once at launch, never again. AI quality isn’t a one-time gate; it degrades silently without ongoing checks.
  • Exact-match testing on open-ended outputs. Punishes correct answers just because the wording differs from the reference.
  • Trusting LLM-as-judge blindly. Powerful, but has documented bias and reliability issues, spot-check it against human review.
  • A golden set that never grows. Real failures should always get added back in, or the same blind spots persist indefinitely.
  • No aggregate tracking. Individual pass/fail per case hides slow, system-wide drift that only shows up when you chart scores over time.

What This Costs In Practice

Building an initial golden set of 30 to 50 cases with rubrics typically takes a few days of focused work. Ongoing regression runs and weekly human sampling are lightweight once the process exists, the cost is mostly discipline, not tooling. Frameworks like LangSmith, Braintrust, and Humanloop handle much of the scoring and version-tracking infrastructure if you don’t want to build it from scratch.

For businesses running AI features on their website or in their operations, booking assistants, AI chatbots, lead qualification, document processing, this is the difference between a system that’s demo-ready and one that’s actually dependable in production. If you’re weighing whether an AI feature needs this level of rigour before you build it, our guide on what to automate first is a useful starting point, alongside our breakdown of real AI agent costs for UK businesses.

Next Steps

If you’re running or planning an AI system and don’t yet have an evaluation process behind it, start small: pull 30 real cases, write acceptance criteria for each, and run them before your next change ships. That single habit catches most of the regressions that otherwise reach customers unnoticed.

Need a testing framework built around your specific AI system: golden sets, scoring rubrics, and regression pipelines included? Get in touch and we’ll scope it, or read more about how we approach managed AI systems for ongoing support without retainer contracts, just ticketed work when you need it.

Sources