Building AI Systems That Survive Model Upgrades

A model may need replacing because its provider retires it, its behaviour no longer meets requirements, or another option performs better in your evaluation. Separating model integration from application logic can reduce the scope of that change. It does not guarantee a configuration-only upgrade.

The Model Upgrade Reality

Model and API lifecycles can require planned migrations. Providers publish retirement notices; for example, OpenAI lists deprecations and shutdown dates. Track the exact models and endpoints your system uses. A new version can affect behaviour, supported features, and cost, so test it rather than assuming improvement.

Tight coupling creates a standing cost: model-specific prompt workarounds, SDKs called from deep inside business logic, fine-tuning jobs that only exist on one vendor, and no baseline for “what good looks like.” When the vendor deprecates the ID, you find all of that at once.

A proposed design is to keep business rules in application code, model interaction behind an interface, and prompts under version control. For example, an internal assistant could put retrieval, permissions, and ticket-writing above a LanguageModel interface. Switching providers would then involve an adapter, prompt testing, and any feature gaps. This is an illustrative design, not a claim about a completed Fernside client project.

Abstraction Layer Design

Separate “talk to a model” from “run the business.”

[Tickets, CRM, permissions, RAG]
 |
 v
 Application services
 |
 v
 LanguageModel port (your interface)
 |
 +--------+--------+
 v v v
 Adapter Adapter Adapter
 (OpenAI) (Claude) (Gemini / Llama host)

The port should look like your needs, not like one vendor’s REST resource: messages in, tools in, text or structured data out, plus usage (tokens, latency, model ID). Adapters translate. They are allowed to use provider extras. prompt caching, specific tool protocols. behind a capability flag. The application must not sprinkle if model == "gpt-…" through domain code.

Practical rules:

Pin model IDs in config, not in source. Config per environment (staging vs production) and per task (classify vs answer vs extract).
Keep vendor SDKs at the edge of the process. One module. The rest of the repo does not know which HTTP client you used.
Normalise errors. Timeouts, rate limits, safety refusals, and malformed JSON should become your error types so retry and fallback logic is written once.
Do not over-abstract on day one. Testing two providers can expose mismatches in the interface, but does not prove compatibility with every provider. A grand internal “AI platform” for a single AI agent is delay dressed as architecture.

Retrieval belongs on the same side of the line as the application, not inside the adapter. Chunk IDs, scores, and citations are your data. The model only sees text you chose to send. That is what lets you change AI provider without touching the knowledge base.

Prompt Management

Version prompts alongside the models and tests they were evaluated with. Recheck them when the model changes.

Version them. A prompt ID, a semver or date, and the model(s) it was tuned against, in git (or a prompt store that still diffs in git). “Whatever is in the database” is not a version.
Separate stable policy from model-facing wording. The rule “never quote a price that is not in retrieved text” is a business rule. The sentences that make a given model obey it are a prompt variant.
Allow model-specific variants without forking the app. answer.v12.claude and answer.v12.gpt can share the same tool schema and output contract. If a variant cannot meet the contract, that model does not ship on that task.
A/B only with measurement. Shipping two prompt versions without logging which ran, and without scoring outputs, is not experimentation. It is noise.

Run the existing prompt on the candidate model and compare the results with the baseline. Change instructions where the results justify it. Neither automatic prompt rewriting nor blindly retaining every workaround is a reliable migration method.

Testing for Model Changes

You cannot unit-test your way to “this model is safe.” You need a fixed evaluation set and a habit of running it.

Build a golden set of representative inputs and expected outcomes. A small initial set can expose obvious problems, but no fixed count such as 50–200 proves suitability for every assistant. Include important task types, known failures, access-control cases, conflicting sources, and adversarial inputs, then expand coverage according to risk.

On every candidate model or prompt version, run the set and compare to the current production baseline:

Task success (did it answer from the docs, did the schema validate)
Safety / refusal
Latency and token cost
Regression on the cases production has already failed once

Have a qualified reviewer examine failures and a sample of accepted outputs. Validate automated graders against human judgements. Set task-specific acceptance criteria and explain how the test set supports them; a single overall average can hide a critical regression.

Safe Rollout Strategies

Schedule the change when the responsible team can monitor it and respond to failures.

Shadow mode. Run the candidate on an authorised copy of selected inputs, with actions disabled. Keep outputs separate and apply appropriate data retention. This adds cost according to the second model’s rates and usage; it does not necessarily double the bill.
Canary. Increase live use in controlled stages; 5%, 20%, and 50% are example rollout choices, not universal thresholds. Use a stable user or tenant assignment where consistency matters. Monitor failures, escalations, latency, and cost.
Task-level cutover. Move a task only after the candidate passes its acceptance tests. Choose the first task by error consequences and recovery options, rather than assuming classification is low risk.
Rollback criteria written down. Define thresholds from the baseline and the cost of errors. Keep the old model, adapter, and prompt available for the rollback period where the provider permits it. A config revert is insufficient if the old model has been withdrawn or a tool has already changed external data.

Do this during hours you have people watching. An upgrade can cause API failures or less visible quality regressions, where requests succeed but answers get worse. Logs, alerts, and the rollback switch are the operational layer. not just “the model is wired up.”

Where Fernside Fits

We can help design AI systems with model interfaces, versioned prompts, evaluations, and rollout procedures. For an existing system, we assess the dependencies and migration risks before proposing a change. Ongoing evaluations and cutovers can be scoped within managed systems.

Want an architecture that can change models without a rewrite? Book a discovery call and we will map what is coupled today and what a portable layer would look like on your stack.