In this 49-minute QCon AI New York 2025 presentation, Jake Mannix, Technical Fellow at Walmart Global Tech, asks what it would take to build LLM-based agents with the engineering controls expected of ordinary production software. He frames the problem from his work on agentic middleware for Walmart's more than 20,000 software developers and data scientists, without presenting Walmart production results for the architecture proposed in the talk. His answer is not to discard the Model Context Protocol (MCP), but to put a small protocol-aware layer between agents and MCP servers.
These notes first summarize Mannix's presentation. A separate synthesis section draws practical conclusions from it.
The Problem: Agent Programs Resemble 1975 BASIC
Mannix describes English as the new programming language for agents, but argues that its current engineering model resembles BASIC in 1975: one large program, little encapsulation, and non-deterministic jumps through an LLM loop. Conventional software gained interfaces, versioned libraries, dependency management, tests, and CI/CD; agent systems have not consistently recovered those properties yet.
MCP is still an important advance. Without shared tool servers, every agent team would repeatedly implement integrations for services such as Slack, Jira, or Stripe. MCP lets domain owners design a tool once and lets multiple agents consume it. Mannix stresses that good tool design is real work: mechanically converting a broad, poorly documented OpenAPI specification can expose hundreds of endpoints at the wrong level of abstraction for an LLM.
His criticism is about the consumption model. An MCP client commonly places the concatenated tool names, descriptions, and schemas directly into the model's context. In his analogy, this is closer to importing a library by expanding its source as a macro than calling through a stable interface. It creates several problems:
- The tool author's terminology may not match the agent's domain language.
- Generic schemas expose optional parameters that consume tokens and give the model irrelevant choices.
- Agent authors cannot easily wrap a tool, bind values deterministically, or combine tools behind a task-specific interface.
- A tool description effectively becomes part of the agent's program, even though another team can change it.
- Agent owners remain accountable for eval results without fully controlling the context that drives those results.
Virtual Tools as an Interface Layer
Mannix proposes virtual tools: versioned mappings between an agent and the underlying MCP tools. The agent and server can continue speaking MCP, while a gateway, proxy, or sidecar rewrites the contract in between. This follows a Strangler Fig approach: extend behavior at the protocol boundary instead of replacing existing agents, frameworks, or MCP servers.
For example, an agent could receive a notify_engineering(message, severity) tool backed by a generic Slack post_message tool. The mapping can:
- Rename the tool to match the agent's vocabulary.
- Replace or tune its description, including examples.
- Project a smaller input schema that exposes only relevant parameters.
- Bind the engineering channel or current user ID outside the LLM, rather than asking the model to reproduce an opaque identifier correctly.
- Translate the virtual call back to the provider's original tool call.
This creates interface and encapsulation boundaries. The agent does not need to know whether notification uses Slack or another service, and the Slack team does not need to customize one global tool description for every consumer.
Mannix extends the idea to composition. Configuration could define sequential calls, parallel fan-out and join, or conditionals, producing coarser tools such as search_everywhere. His goal is deliberately limited: add enough deterministic orchestration to a ReAct-style agent without requiring every team to adopt a full workflow engine such as Temporal or move to an LLM-generated-code approach such as CodeAct.
Deterministic Security Instead of Prompted Policy
The same intermediate layer can enforce policy. Mannix uses Simon Willison's lethal trifecta to explain the risk created when one agent context combines:
- Access to untrusted content.
- Access to private data.
- A way to communicate externally.
An email-reading tool, an internal user-data tool, and a publishing tool are one example. A malicious instruction in untrusted content could induce the model to read private data and send it out. Mannix also notes that a generic URL-fetching tool can both ingest untrusted content and encode data into outbound requests, so dangerous combinations are not exotic.
His core security claim is that system-prompt guardrails are not an enforcement boundary. The model is probabilistic, is intentionally influenced by its inputs, and can be attacked. Policy therefore needs deterministic checks outside the model.
He proposes taint tracking based on tool metadata. Tools would be required to classify capabilities and data flows such as returning personally identifiable information (PII), accepting untrusted input, or reaching the external web. MCP already has advisory tool annotations for properties such as read-only, destructive, and idempotent behavior; Mannix proposes making relevant annotations enforceable within an enterprise and extending the metadata needed for data classification. The middleware propagates classifications through the active context and applies policy at several stages:
- Registration or build time: reject a disallowed tool set, such as open-web access in a HIPAA-regulated workflow.
- Review time: require explicit approval for a risky but potentially valid combination.
- Runtime: after a call introduces PII into the context, use MCP's
notifications/tools/list_changedmechanism to remove tools with open-world access before the model's next turn.
Removing a capability before the model selects it is preferable to allowing a prohibited call and then failing it. Mannix acknowledges that a changing tool list can disrupt an earlier model-generated plan, but argues that the attack surface is still reduced because the dangerous combination cannot form.
A Registry Makes Agent Dependencies Operable
Virtual-tool definitions, policies, and taint rules need a durable home. Mannix proposes an enterprise registry that acts as part package registry, part build system, and part deployment gate. He does not prescribe whether this should be called an MCP, A2A, or broader AI registry.
He is especially critical of relying on an unversioned, dynamic MCP tools/list in production. If a server silently changes a tool's schema or description, the agent's behavior and eval results can change without a corresponding application release. His proposed alternative is to pin a virtual tool version, such as notify_engineering 1.3, to an immutable definition.
The registry would also hold or reference:
- Typed input and output schemas, using MCP's existing output-schema support rather than treating every result as an opaque string.
- Dependency graphs connecting agents, virtual tools, MCP servers, and services.
- Agent cards containing identity, API location, schemas, LLM configuration, security policies, and references to evals.
- Cached configuration distributed to protocol proxies, avoiding a registry lookup on each tool call.
This concentrates non-determinism in the model rather than adding avoidable uncertainty through mutable dependencies.
Eval Packs Turn Examples into Contracts
Mannix's registry proposal also includes references to eval packs. An eval pack consists of:
- An eval dataset containing example inputs and expected outputs that conform to the agent's schemas.
- An eval runner, referenced as an executable, that invokes the registered agent through the endpoint in its agent card.
- Evaluation configuration that compares actual and expected results, potentially using an LLM judge.
He calls these examples an "inductive PRD": instead of describing requirements only in prose, a product owner supplies representative input-output pairs. This makes basic evaluation accessible to people who can define desired behavior but are not data scientists.
Central registration also enables other teams to test changes without modifying the owner's canonical dataset. Security can clone and adversarially transform cases; finance can compare a cheaper model using the agent's own quality criteria; a data science team can evaluate a fine-tuned model across registered agents. These are proposed organizational benefits, not performance results reported from a deployed implementation.
What Is Implemented and What Is Proposed
Mannix is explicit about the maturity of the idea. MCP registries and proxies already exist, and he says he produced a functional prototype of the proposed virtual tools, combinators, and taint tracking over a weekend or two using Claude Opus. He also says the prototype has not demonstrated production performance or scale. Virtual tools, policy-aware rewriting, and the broader registry model should therefore be read as an architectural proposal backed by an experiment, not as a completed Walmart production case study.
He presents a spectrum rather than one mandatory architecture: lightweight configuration and virtual tools at one end, workflow engines in the middle, and CodeAct-style generated orchestration at the other. These approaches can coexist. His immediate aim is to recover basic encapsulation, interfaces, and deterministic controls without waiting for a more ambitious agent runtime.
Synthesis: Practical Lessons
The following conclusions are my synthesis from the presentation, not direct claims by Mannix.
Treat tool metadata as executable code
A tool name, description, and schema influence model behavior, so changing them should trigger the same controls as changing a library dependency: review, versioning, compatibility checks, and regression evals. Recording only the MCP server endpoint is insufficient for reproducibility.
Give the model the smallest useful interface
Schema projection is not merely a token optimization. Binding tenant IDs, channels, permissions, and user identity outside the model removes decisions the model should never have owned. This improves reliability while creating a clearer authorization boundary.
Separate capability from policy
Tool providers should accurately describe capabilities and data classifications; agent owners should define task-specific interfaces; security teams should define enforceable combinations. A protocol-aware layer can connect these responsibilities without forcing one team to own all three.
Preserve evidence alongside deployment configuration
Pinning tools without pinning evals proves reproducibility but not quality. Linking immutable tool definitions, model configuration, policy, and eval packs provides a stronger release record: what ran, what it could do, and what evidence justified deployment.
Start with static controls
Runtime taint tracking is powerful but introduces planning and state-management complexity. A practical adoption path is to inventory tools, require typed schemas and classifications, reject obviously unsafe combinations at registration, and only then add dynamic capability removal where static policy is too restrictive.
Key Takeaways
- MCP reduces duplicated integration work, but direct exposure of mutable tool definitions leaves agent owners without ordinary interface and dependency controls.
- Virtual tools adapt provider-owned capabilities into agent-owned, narrow, versioned interfaces.
- Security policy should be enforced outside the LLM; prompts are guidance, not a deterministic authorization mechanism.
- Proposed data classifications, existing MCP annotation hints, and taint propagation can support deterministic controls at build, review, or runtime.
- A registry can connect pinned dependencies, schemas, policies, dependency graphs, model configuration, and eval evidence.
- Mannix's virtual-tool and taint-tracking implementation is a prototype, not a demonstrated production-scale system.
Canonical source: Jake Mannix, "From Copy-Paste to Composition: Building Agents Like Real Software", QCon AI New York 2025 presentation and slides, published by InfoQ on July 22, 2026.