The AI Gateway: Scaling Centralized Inference across Decentralized Teams

The talk's thesis fits in one sentence: model choice should be decentralized, but model access must be centralized, and an AI model gateway is the thin layer that reconciles the two. Meryem Arik, co-founder and CEO of Doubleword (previously TitanML), argues that organisations now run many models from many providers, and that letting each product team wire its own API keys straight to each provider produces an ungovernable mess — while the obvious cure, mandating a single provider, produces worse applications. The gateway is her answer to that tension, and she is unusually direct that she is trying to persuade every listener to deploy one.

This is a 46-minute QCon AI presentation, dated 20 May 2026 in the InfoQ page's "Recorded at" metadata. The problem it addresses is what Arik calls inference chaos, and her origin story explains where it comes from. Doubleword was founded around four years ago focused entirely on inference — the process of actually running models to produce output, as opposed to training them. What she kept observing was that Doubleword was never her clients' only inference provider: the same organisation would be using Doubleword plus OpenAI plus perhaps Mistral plus self-hosted fine-tuned models they had built themselves. A multi-provider estate is the default state, not an edge case, and the chaos follows from it. Her company ended up building an open-source gateway to fix this for clients, which is the second disclosure worth flagging: gateways are not Doubleword's commercial product, but Doubleword ships one and she includes it in her recommendations. The talk was inspired by a blog post on control layers written by her CTO, whose blog she points listeners to at https://fergusfinn.com/blog/control-layer/. Points marked Context below are supplementary explanation for readers unfamiliar with the terminology, not claims made in the presentation.

What You Will Learn

• Why a single model cannot serve every use case, and the three dimensions teams actually use to choose one.
• Why centralizing inference matters for different reasons depending on whether you self-host or buy tokens.
• What an AI model gateway does on every request, and how that differs from a traditional API gateway.
• Why Arik argues the gateway should be deliberately "dumb" — stateless, low latency, no smart routing — and who disagrees.
• How RBAC, budgets, and rate limits are modelled through groups imported from your identity provider.
• Where the gateway concept is heading: agent gateways and MCP gateways.
• Which failure modes and blind spots the pattern does not solve, including natively embedded assistants like Copilot.

Why Decentralized Teams Need Different Models

Arik opens with a hunting analogy that survives translation better than most: a successful hunt uses pointers to locate game, spaniels to flush it out, and retrievers to bring it back. No single dog does all three well. Likewise, a single application often needs several models cooperating, and different applications need entirely different ones. There is no model that rules them all.

She decomposes model selection into three dimensions, and it is worth treating this as a checklist rather than a list of opinions.

Application quality is how well the model actually solves your problem. Sub-dimensions she names are modality (embeddings, image, voice, and text are different model families), general capability measured through benchmarks and Elo scores, and domain fit. She cites healthcare clients who need models that understand their specific domain — labelling cancer screens is her example of a very task-specific job.

Non-performance reasons are, in her words, the most boring and the most binding. Existing vendor pre-commits force your hand: if you have spent credits with a cloud provider, you use what that provider hosts. Data residency requirements under GDPR or local law can restrict you to models served from a particular region or by a particular compute provider. These constraints narrow the candidate set before quality is even considered.

Note on a transcript ambiguity. The transcript renders her AWS example as "if you're an AWS shop, you probably are going to be using OpenAI because everything you want needs to be within AWS." OpenAI's models are not the canonical AWS-native choice, so this is likely a misspoken or mistranscribed vendor name. The point is unambiguous and is what matters: your cloud commitment constrains which models you may realistically use.

Inference performance covers cost, latency, throughput, and rate limits. Her framing here is the most transferable idea in the section: acceptable cost is a function of what you charge, not an absolute number. A consumer chatbot billed at $5 per month per subscriber needs very cheap inference. A cancer-screening product with high per-user revenue can absorb far more expensive inference. She also notes that latency requirements push you toward specific providers — Cerebras and Groq for very low latency, others for high-volume work (the transcript renders the second as "Grok"; in context she means Groq, the LPU vendor, not xAI's model) — and that some workloads can run asynchronously, which relaxes the latency constraint. Throughput and rate limits belong on this list as selection criteria, not only as governance knobs: a provider whose rate limit sits below your required request volume is disqualified before any quality comparison happens.

Context. Cerebras and Groq build custom inference silicon (wafer-scale engines and LPUs respectively) that serve tokens at much lower latency than general-purpose GPUs, generally at a different cost profile. Arik notes her own company optimises for high-volume workloads rather than low latency, which is why she names others for that job.

Three Worked Examples

She grounds the framework in three use cases whose requirement profiles barely overlap.

The guardrail row is the one most teams underestimate. Because a guardrail model is invoked on every request before the main model, its latency is added to everything, and its data-handling posture applies to everything.

Why Inference Must Still Be Centralized

Taken to its natural conclusion, full decentralization produces what Arik calls spaghetti: every team calling many providers directly, API keys scattered everywhere, nobody able to answer basic questions. Her list of the resulting symptoms is concrete — twelve use case teams, eight models, three or more providers, thirty-seven API keys generated for reasons nobody remembers, products nobody approved, and an intern who spent $4,000 over a weekend. She adds that she has seen this in real life and the real number was much higher.

Her historical point sharpens the argument. Eighteen months ago this genuinely was fine, because one team was experimenting and everyone used OpenAI since it was clearly the best model available. That is no longer true, so the sprawl that was previously theoretical is now the default state.

The reasons to centralize differ by deployment model, and she splits them deliberately.

If you self-host, the dominant reason is GPU utilization. GPUs are extremely expensive, and five teams independently deploying the same open-weight model is straightforwardly wasteful. Centralization also lets you smooth load across use cases, which raises utilization further, and lets you monitor reliability and uptime — she notes self-hosting means operating a much more fragile ecosystem than calling a hosted API.

If you buy tokens, the reasons are commercial and, she concedes, boring: you can negotiate bulk discounts, sign specific data retention policies, and secure higher rate limits. None of those are available to a team acting alone with its own key.

Regardless of deployment, she lists access policies (a model fine-tuned on customer data should not be reachable by an intern), auditability of what went into and out of every model, cost controls, reliability and uptime monitoring, company-wide guardrail policies, and data controls supporting locality requirements.

The takeaway she states explicitly: centralization at inference time is essential for governance and cost optimisation — but it must not become the crude version of centralization, which she characterises as decreeing "we're only using AWS Bedrock now" and accepting worse applications as the price.

What an AI Model Gateway Actually Does

The gateway is the layer that sits between every application and every model. Arik draws the analogy to traditional API gateways — illustrating it with Kong, on the assumption that most of the room already runs one — and then immediately qualifies it: an AI model gateway is meaningfully different, because AI requests have features normal API requests do not. You want to monitor different things — she does not enumerate them, but Context: tokens consumed, cost per call, and which model served the request are the obvious examples an API gateway has no concept of. You may also want to enforce guardrails, which has no API-gateway analogue, and you need model-aware routing.

The capabilities she enumerates:

• Unified API access. This one benefits the use case teams more than the central team. Providers use slightly different request schemas, so swapping models without a gateway means re-plumbing code; with one, it is a config change.
• Access controls. Especially important for self-hosted models, because serving stacks do not provide them. Arik names vLLM directly, stating that it "doesn't come with authentication or authorization," so something must supply them. (Context: vLLM is a widely used open-source inference server for self-hosting LLMs. It does offer a single static bearer token via --api-key, so her statement is best read as meaning it has no identity-aware authn or per-user authorization — no users, groups, or RBAC. A vLLM endpoint left on an internal network with a shared token or none at all is a common and serious misconfiguration.)
• Logging, monitoring, and auditability. Two motivations she gives are worth separating. The obvious one is answering "why did we make that decision" twelve months later. The less obvious one is that captured request data becomes your fine-tuning corpus later.
• Model routing. Two triggers she names: routing on request difficulty, and routing on load. The failover case is the most immediately practical — when a provider has an outage, automatically shifting traffic to another provider avoids inheriting their downtime.
• Cost controls and rate limits, applied by group.
• Guardrails, failovers, and unified prompt management.

Architecture And Data Flow

Every request from every application passes through the gateway. Arik describes the request carrying a prompt plus metadata, and the metadata content is the part worth copying: application context, the SLA required (real time versus tolerable ten-minute latency), data handling requirements such as deletion deadlines, and routing requirements including which failovers are acceptable. The gateway performs logging, monitoring, authentication, and API key checks, forwards to the chosen model, and returns the response. The data it accumulates then feeds chargeback reports, alerting, and user ratings.

flowchart TD
    subgraph Apps[Decentralized use case teams]
        A1[Coding assistant]
        A2[Data labelling pipeline]
        A3[Customer-facing app]
    end
    A1 --> GW
    A2 --> GW
    A3 --> GW
    GW[AI model gateway
minimal work, low latency]
    GW --> AUTH[Auth and API key check]
    GW --> RBAC[Group-based access control]
    GW --> BUD[Budgets and rate limits]
    GW --> LOG[Request logs, tokens, cost]
    GW --> ROUTE{Routing and failover}
    ROUTE --> SH[Self-hosted models
vLLM, fine-tuned]
    ROUTE --> BATCH[Batch / ASIC inference
her own company's offering]
    ROUTE --> P1[OpenAI]
    ROUTE --> P2[Anthropic]
    SH --> GW
    BATCH --> GW
    P1 --> GW
    P2 --> GW
    LOG --> STORE[(Request store)]
    STORE --> CB[Chargeback reports]
    STORE --> AL[Alerting]
    STORE --> FT[Future fine-tuning data]
    IDP[Identity provider
e.g. Entra ID] -->|import groups| RBAC

Because the gateway sits on every request, Arik insists it must be very low latency and effectively invisible to development teams — a principle she develops further below.

She is also careful to frame the payoff as a benefit to both sides rather than a compromise between them. Decentralized use case teams innovate faster because nothing is restricted, they keep access to every model and tool they care about, and they can swap models without re-architecting around a new provider's schema. The central inference team gets consistent access control and governance, plus protection from runaway spend, without imposing any of those limitations on the teams. That is why she calls it a no-brainer.

The Gateway Options She Named

She surveyed five, noting the first four are open source and stating a personal preference for open source specifically so you can self-host the gateway itself.

She also mentions that vendor-specific gateways exist — Databricks and Microsoft both offer one — and says she deliberately did not recommend them, because she philosophically believes this layer should be independent of the underlying infrastructure. The reasoning is portability: if your gateway is welded to Databricks, migrating to Snowflake means rebuilding your control layer.

Note that the performance ranking is her own assessment of her own project. She does supply a number, discussed below, but it is a vendor-run benchmark.

Configuring Access: Groups, Budgets, and Rate Limits

Walking through gateway UIs, she highlights three screens.

The model catalogue is what a use case developer sees: the models they have access to, filtered by their roles and groups, mixing self-hosted and API providers. Her walkthrough example is a developer picking GPT-5 Nano, generating an API key for it there, and starting to build. Context: this self-service issuance is the direct answer to the thirty-seven-orphaned-keys problem: keys still get created on demand, but every one of them is minted by the gateway, so the central team knows all of that traffic flows through it. She notes the catalogue is also a genuinely useful discovery mechanism, since teams frequently do not know what is available.

Request logs capture every request, which model served it, latency, and cost. She flags these as sometimes dangerous and warns you must be careful about who gets access, since the logs contain prompt content. The upside is that this is what lets you verify you are not violating policy.

Groups are the unit of permission. She believes most providers have standardized on groups as the mapping between users, models, and permissions — so a fine-tuned model trained on PII can be restricted to named groups. Budgets and rate limits attach to the same groups. Her illustration: an intern group capped at a $15 budget with a deliberately low rate limit because the project is low priority, versus a mission-critical flagship team with a very high budget because the use case must always work. Requesting access to a model becomes a step in the team's onboarding rather than an unchecked act.

Answering a question about enterprise identity, she was firm about where the source of truth lives. The platform team — typically an AI platform team sitting in the CTO's office — owns the gateway, but groups should be imported from the identity provider such as Entra ID rather than invented in the gateway. All authentication and group membership stays in the IdP; the gateway only maps those groups to permissions. This is consistent with her "do as little work as possible" principle. She confirmed separately that these gateways are enterprise-ready in the sense that matters here: most providers, hers included, have SSO integrations. Her qualifier ties this back to her open-source preference — federation works best "especially if you're using one of the self-hosted ones where you can deploy it in your own environment and then they'll connect to your SSO." A gateway you run inside your perimeter is the one that can sit behind your existing identity provider rather than beside it.

Why the Gateway Should Be Dumb

This is the talk's most opinionated position and the one Arik defends most consistently.

Today's gateways are declarative: a team says which model it wants. An obvious evolution is smart routing, where the gateway itself decides the best model per request — which is effectively what consumer products like ChatGPT and Claude do internally, constantly changing models behind the scenes. Some providers offer this; she names Not Diamond, hedged with an "I think," as a company doing intelligent routing.

She is reluctant to advocate it for two reasons. First, latency: any work the gateway does is added to every request, so the gateway should do as little as possible. Second, autonomy: use case teams need the freedom to design the experience their users need, and hiding model selection removes it.

Transcript correction. As transcribed, her closing sentence on this point reads "the model gateway should be smarter," which directly contradicts every other statement she makes. Context makes the intended meaning unambiguous — she is arguing the gateway should be dumber and the use case teams should have more freedom.

Asked what patterns keep it that way, her answers were specific:

• Avoid state wherever possible. The questioner's framing was correct and she agreed that state is what makes horizontal scaling hard; her own phrasing was the hedged "we try to not keep it stateful if we can," not an absolute guarantee of statelessness.
• Optimise the implementation. Doubleword wrote theirs from the ground up in Rust, which she credits for its efficiency, and the reason they built their own rather than adopting LiteLLM was that LiteLLM's latency was too high for their scale.
• Do not force optional work into the path. She specifically cites not mandating guardrails, because guardrails add substantial latency.
• Include only what is universal. Their design rule was to implement what is replicable across every company and application, and let everything else be built on top.

On concrete numbers, she said gateway overhead should be essentially zero, and referred to internal benchmarking against Bifrost and LiteLLM that was about a month old at recording time. As she described it, on a hardware setup she did not detail, the requests-per-second figure was around 200, while LiteLLM managed roughly 60 because it added a lot of overhead. The transcript does not attribute the 200 figure to any named product; reading it as Doubleword's own gateway is an inference from the surrounding argument rather than something she says. She immediately qualified the comparison — LiteLLM has since improved, and it was only slow for them because of their scale; she knows many people running LiteLLM happily at smaller volumes. Treat these figures as a vendor's self-reported comparison of an older version, not an independent result.

She is equally clear on the flip side of the "dumb" principle, in answer to a governance question. Doubleword's gateway does not integrate guardrails, because she considers guardrails and governance highly application- and use-case-specific and therefore better handled outside the gateway. Portkey takes the opposite approach, and she presents this as a legitimate choice rather than a mistake — "you can have it built in or not have it built in." Her practical bridge is that most gateways are OpenTelemetry-native, so they plug straight into Datadog or whatever observability stack you already run.

Where This Is Going: Agent and MCP Gateways

Arik was explicit that the model she presented — the gateway handling simple LLM calls — is already becoming outdated. As use cases grow more complex, more happens at inference time, and she expects the same linchpin to absorb it.

An agent gateway is the first extension. Instead of calling an LLM, a team increasingly calls a whole agent to perform an entire sub-task. That, she argues, suffers from exactly the same sprawl and governance problem, so it belongs behind the same control layer.

An MCP gateway is the second, and here she leads with security. She does not want to tell teams to use any MCP server they like, because of the security risks involved; the alternative is a pre-approved set that teams can use freely while the organisation retains control. She noted these projects are mostly less than twelve months old and are already evolving in this direction, and expects something further in another twelve.

Context. MCP (Model Context Protocol) standardises how models connect to external tools and data sources. The risk she alludes to is that an MCP server is arbitrary third-party code granted access to your model's context and often to real systems — so an unvetted server is a supply-chain and data-exfiltration exposure, not merely a dependency.

An audience member pushed on scale: a company might allow a few hundred models but could plausibly have tens of thousands of agents, which seems unmanageable. Arik's reply was twofold. Rhetorically: "isn't it more messy to not have it?" Substantively: central teams are not expected to build the agents. Use case teams build their agentic workloads and then register them with the gateway. What she sees in practice is a request-based system, and her example is concrete: a team says it wants to use the new Qwen model and asks whether it can be registered via Bedrock; the platform team registers it; the team receives an API key.

Trade-offs And Limitations

Externally-facing gateways need surprisingly little extra. An attendee whose customers interact with their AI gateway directly — via published MCP servers behind a trusted public endpoint, connected to products like Teams or Google Agentspace — asked whether that changes the design. Arik's answer was essentially no. Such an organisation already has a gateway, because that is how key distribution works. If you accept her school of thought that the gateway should be stupid and robust, there is no external-versus-internal-specific logic to add, because the gateway concept is the same either way. Her reframing is neat: a centralized inference team can think of its job as serving externalized use case teams.

Natively embedded assistants are largely out of reach. Asked about Copilot inside Microsoft Office, Arik conceded that when a model is natively embedded, gateway-level visibility is very difficult, because what you can see is entirely up to what Microsoft chooses to show you. She offered two responses rather than a fix: it is a reason to favour providers that grant that access control, and Microsoft does provide other governance mechanisms, so accepting the trade may be reasonable. Either way, your gateway will not give you unified cost attribution across embedded assistants.

A gateway on every request is a new single point of failure. Arik does not frame it this way, though her "dumb gateway" principles above all reduce the exposure. Context: the architecture nonetheless concentrates risk, since an outage or latency regression in the gateway degrades every AI feature at once. The talk does not discuss gateway high-availability, so treat it as your own design problem.

Request logs are a liability as well as an asset. She raised this herself. The same logs that give you auditability and future fine-tuning data contain prompt and response content, which may include customer data or PII. Access control on the logs is a separate problem from access control on the models.

Dumb gateway versus smart routing is a genuine disagreement. Arik presents two schools of thought and picks one, for the latency and team-autonomy reasons given above. Context: the counter-argument she does not develop is that routing easy requests to cheaper models is a direct cost lever, so a cost-dominated batch workload weighs her two objections very differently than an interactive one does.

Reusing your existing API gateway usually fails. Asked about adapting something like Apigee, she said customers try this and typically decide against it. The convenience features are the difference: prompt management, native support for new provider APIs as they ship, and the expected expansion into MCP, vector database, and agentic gateways. Her conclusion is that the divergence is large enough to justify a separate component.

Model quality is not a property of the model alone. In response to a question about how much of output quality is the model versus the prompt, she gave what is arguably the most useful correction in the Q&A: think in terms of prompt–use case–model triplets, like a racehorse and its rider. Asking a model to write Shakespeare while prompting for Dickens will fail, and that is not the model's fault. The practical consequence is that you should never write off a model as bad for your use case; you write it off as bad with that prompt for that use case. This is also why she says use cases are hard to build centrally: iterating the prompt requires domain-specific knowledge that lives in the use case team, not the platform team.

Vendor perspective. Arik disclosed her interests clearly and repeatedly: gateways are not Doubleword's commercial product, but Doubleword ships one, she recommends it alongside competitors, and the performance claims supporting it are her own. Note a tension the talk does not reconcile. Early on she says "we don't sell it, it's not a commercial project of ours"; later, answering a question about enterprise readiness, she says "our product is not building gateways — we actually built this for our enterprise clients and so it's been in production with them for a while, just incidentally." Both can be true if the gateway ships alongside a paid inference product without being separately licensed, but read the recommendation with that in mind. The structural argument for centralization stands independently of which gateway you pick.

Practical Takeaways

Arik closes on the same three-part spine she announced at the start, and it is worth holding onto as the argument's skeleton: teams need the freedom to pick the right tools for their use case, or they will build worse applications; centralization at inference time is essential for governance and cost optimisation; and an AI model gateway is the tool that delivers the second without sacrificing the first. Everything below is a way of acting on that.

• Deploy a gateway even at small scale. Her explicit claim is that this is genuinely less than half a day of work, lightweight, and cheap to host. If your organisation has more than one team calling more than one provider, the spaghetti has already started.
• Start with an open-source option you can self-host. LiteLLM is the fully-featured default; Portkey if you want built-in guardrails; Doubleword or Bifrost if performance dominates. Avoid gateways welded to a specific data platform so you retain the ability to migrate.
• Import groups from your IdP; do not invent them. Map Entra (or equivalent) groups to gateway permissions rather than maintaining a parallel identity model. Budgets and rate limits then attach to groups you already govern.
• Set budgets and rate limits per group before you need them. The $4,000 intern weekend is preventable with a $15 cap and a low rate limit, and costs nothing to configure in advance.
• Put the SLA, data-handling, and routing requirements in request metadata. This is what lets the gateway make correct decisions without becoming smart.
• Keep state out of the gateway and optional work off the critical path. Guardrails, if you use them, should be an application choice rather than a mandatory hop for every request.
• Evaluate models as prompt–use case–model triplets. Rejecting a model on a single prompt's results discards information you have not actually gathered.
• Pre-approve MCP servers now. The security exposure exists whether or not you have a control layer; a pre-approved list preserves team velocity while bounding the risk.

Key Terms

• AI model gateway — a control layer that all model requests pass through, providing unified API access, authentication, RBAC, logging, routing, budgets, and rate limits across multiple model providers.
• Agent gateway — the same control layer extended so that teams can call whole registered agents, not just individual model completions.
• Chargeback report — an attribution of AI spend back to the team or use case that incurred it, generated from gateway request logs.
• Control layer — the general term for a thin, centrally-owned component that mediates access to a shared resource; the concept the talk is built on.
• Elo score — a relative ranking derived from head-to-head comparisons, used to rank model capability against other models rather than an absolute scale.
• Groups — the standard unit that gateway providers use to map users to model permissions, budgets, and rate limits, ideally imported from an identity provider.
• Guardrails — a model or rule layer that inspects requests and responses for policy violations before they reach the main model or the user; latency-critical because it sits on every request.
• MCP (Model Context Protocol) — a standard for connecting models to external tools and data; an MCP gateway restricts teams to pre-approved servers.
• Model-aware routing — selecting a target model per request based on difficulty, load, or provider availability, including automatic failover.
• Data residency — a requirement that data be stored and processed within a specific jurisdiction, which constrains which models and providers are usable.
• vLLM — a popular open-source LLM serving engine for self-hosting, which ships without identity-aware authentication or per-user authorization, offering only a single static API key.

Reference: The AI Gateway: Scaling Centralized Inference across Decentralized Teams