Route models through a gateway, not an SDK
Calling a provider SDK directly puts model choice in your application code. A thin gateway makes it configuration, which is what you need the day a model is deprecated.
Most LLM features start with a provider SDK imported directly into the service that needs it. That is the right call for the first version. It becomes the wrong call at roughly the point where you have more than one model, more than one tenant, or a finance conversation about token spend.
What direct SDK calls actually couple
The import is not the problem. The problem is everything that accretes around it: retry policy, fallback, token accounting, prompt-cache handling, and the model name itself all end up as application code, duplicated per call site with small inconsistencies.
Then a model is deprecated on a date you do not control, and the change is a codemod across every service instead of a config edit.
What a gateway is and is not
A gateway here means a thin internal layer that every model call goes through. Not a product, not a proxy service you operate separately unless you need to - a library boundary is often enough. It owns four things:
- Model selection, resolved from configuration rather than a literal in code
- Fallback order when a provider degrades or rate-limits
- Token and cost accounting, attributed to a caller rather than a total
- Cache strategy, so the split between cacheable instruction and per-request context is decided once
The application asks for a capability - summarise, classify, plan - and the gateway decides which model serves it. That inversion is the whole point.
Where it earns its keep
Per-tenant routing becomes possible: one brand on a frontier model, another on something cheaper, with no branching in the feature code. Cost attribution becomes per-node rather than a single monthly number, which is the difference between knowing you spend too much and knowing which step is doing it.
If you cannot answer which agent step costs the most, you do not have a cost problem yet - you have an attribution problem.
The honest cost is one more indirection to debug through, and a real risk of over-abstracting before you have two models to route between. Build it when the second model arrives, not in anticipation of it.