Writing

Access gateways and serving gateways are not the same thing

Most AI gateway discussions I see still assume one architecture: a reverse proxy that routes outbound calls to whichever model provider is cheapest or fastest. That assumption starts to crack the moment your team actually exposes a proprietary model to paying customers. The moment inbound customer traffic arrives, the questions change. You are no longer optimizing someone else's API call; you are responsible for who can call your model, what they are allowed to do, how much they consume, and what record you keep of it.

Baseten's breakdown of the two patterns separates them into access gateways and serving gateways. Access gateways handle outbound routing across providers: selection, fallback, caching. If one provider is slow or too expensive, the access gateway moves traffic to another. Serving gateways sit in front of your own models and manage inbound customer traffic: identity, routing, abuse protection, metering, audit logs. Both are called gateways, but they are built for different sides of the same transaction.

The practical point I keep coming back to is that if you are operating a model commercially, the serving gateway is not an optimization layer. It is the control surface for revenue and compliance. A poorly tuned access gateway might cost you some margin on outbound inference. A poorly built serving gateway can mean customers get access they should not have, billing is wrong, or you cannot demonstrate to an auditor what happened. Those failures are not performance problems; they are business problems.

The serving gateway isn't an optimization layer; it's the control surface for revenue and compliance.

One thing I will watch for is teams that invested early in access gateways. They often assume they already have gateway coverage. They do not. An access gateway is designed to make outbound calls cheaper and more reliable across a set of external providers. It does not know which customer is behind a request unless you build that in, and it does not produce per-tenant metering records by itself. When those teams begin serving a proprietary model, they may point their existing proxy at their own endpoint and think the problem is solved. It is not. They have an outbound routing tool in front of an inbound revenue service.

Metering and per-tenant routing require a different posture entirely. You need to attach identity to every request, enforce quotas and rate limits, flag abuse signals, and keep audit logs that hold up under scrutiny. Those are not features you can bolt onto an access gateway with a middleware rule or two. They sit at the core of how the gateway is designed. An access gateway can be fairly stateless; a serving gateway has to maintain tenant state, policy, and billing context on every call. The design constraints are different enough that it is usually cleaner to operate a dedicated serving gateway than to stretch an access gateway into a role it was never meant to play.

I expect serving gateways to become a default component in AI product stacks, not a bolted-on afterthought. The moment a model is behind a commercial interface, the serving gateway is the piece of infrastructure that turns raw inference into a managed service. Teams that recognize the difference early will spend less time retrofitting tenant controls later.

Originally covered at baseten.co ↗