Cypher-X

Aaron Erickson's central argument is not that reliable systems must choose between deterministic software and AI agents. It is that they need both, with a deliberate boundary between them: agents are useful for discovery under uncertainty; tools are useful when an action must be predictable.

Erickson founded the Applied AI Lab for DGX Cloud at NVIDIA. In this QCon San Francisco 2025 presentation, published by InfoQ on July 7, 2026, he draws mainly on his team's experience building LLo11yPop, a large-language-model system for GPU observability. The 53-minute presentation is a practitioner account, not a controlled benchmark. Its measurements and examples should therefore be read as reported project experience rather than general performance guarantees. Except for the explicitly labeled synthesis section, these notes summarize Erickson's claims and examples rather than independently validated findings.

What You Will Learn

Why constraining an agent's role, data access, and tool choices can improve reliability.
How retrieval, analyst, orchestrator, action, and tool agents were separated in LLo11yPop.
Why internal vocabulary and operating knowledge are "rare context" that a generic model cannot infer reliably.
Where deterministic queries, runbooks, calculators, and transaction services fit inside an agentic workflow.
How to structure agent evaluations like a test pyramid while accounting for accumulated error and inference cost.
Why permissions, grounding frequency, human review, and feedback quality are platform concerns rather than prompt details.

The Project Behind the Lessons

Before NVIDIA, Erickson's startup built an event-based organizational-design product. In 2023, the team connected it to an early ChatGPT plugin. The model could read an organization represented as JSON, select from nine kinds of personnel moves, write Python to propose a restructuring, submit those moves as a draft, and generate the accompanying email.

The example established both sides of the talk's argument. Models can combine data, code, and actions into a surprisingly capable workflow. The consequences of acting on that workflow, however, are too important to entrust to unconstrained generation.

At NVIDIA, Erickson worked on a system for allocating scarce GPUs among research groups and observing cluster health. The domain included cloud providers, regions, tightly networked GPU blocks, training jobs, idle capacity, and signals such as fan failures, temperature, and power fluctuation. The team initially tried to apply an LLM broadly, but found that accuracy improved when each model received a narrower problem.

That work became LLo11yPop, short for "large language model for observability," with "pop" appended to make the name a word. Erickson says the team produced it after approximately four or five months. Later, after the team had moved on and MCP and agent frameworks had matured, another engineer rebuilt what Erickson characterized as the same thing in about six hours with MCP, LangChain, and other available tooling. This is his anecdote about reduced implementation effort, not a documented equivalence test between the two systems.

Separate Retrieval, Analysis, and Action

LLo11yPop used a hierarchy of purpose-built components instead of asking one model to understand the fleet, query every system, diagnose a problem, and act:

Retrieval agents translated a question into SQL, an Elasticsearch query, or a particular API call. A retrieval agent generally knew one query or one table and received a few good examples.
Analyst agents knew the diagnostic questions for a narrow subject, such as patterns associated with H100 heat failures. They did not need database mechanics; they called retrieval agents for evidence. The team used prompt engineering and RAG initially, then some fine-tuning.
An orchestrator agent pursued a goal by consulting several analyst agents and combining their findings.
Action agents prepared recommendations for operational channels such as Slack or Jira. In Erickson's account, the system did not autonomously operate the entire GPU fleet.
Tool agents performed narrow integrations, such as creating a Jira ticket in a known format.

The division matters because context and choice are finite budgets. A broad agent can spend tokens deciding among many nearly equivalent tools, attempting joins it need not invent, or reasoning about irrelevant questions. A specialist can instead receive the examples, schema, and permissions needed for one job. Erickson calls the broad menu of choices a "paradox of choice" and recommends a hierarchy that routes work to a small relevant set.

The analogy to an organizational hierarchy is intentional but limited. A higher-level agent has broader context and coordinates goals; lower-level agents perform specific work. Erickson warns against over-anthropomorphizing models. The useful property is decomposition around limited context, not imitation of human job titles.

This is not an argument for splitting a system into the smallest possible agents. Erickson compares composed, specialized models with microservices: the specialization can work well, but it brings similar coordination complexity and tradeoffs. The appropriate decomposition is purpose-built, not maximally atomic.

Rare Context Is Part of the Platform

Operators asked LLo11yPop questions such as, "Where are the zombie nodes?" A generic model could form a plausible interpretation, perhaps looking for nodes without network connectivity and consulting DCGM metrics, but it did not know what NVIDIA's operators meant by "zombie." Erickson calls company-specific language and examples such as this rare context. Related context that his architecture supplied included:

Company-specific vocabulary and abbreviations.
Local definitions of operational states.
Historical incidents and the patterns experts learned from them.
Known query forms and runbooks for internal systems.
The location and trustworthiness of relevant data.

This is why he rejects claims that an agent can enter any company and work reliably zero-shot. General model knowledge does not contain each organization's language and tacit operating rules. The platform must capture that context. In the examples from the talk, the team did so through few-shot examples, prompt engineering, RAG, and later some fine-tuning; tools and constrained retrieval made the context usable.

Context quality matters as much as context quantity. Erickson notes that a wiki, SharePoint site, or database can itself contain incorrect statements. Retrieving from an internal source does not turn it into ground truth. He favors vetted, organized data products whose owners have already invested in factual quality.

Use Determinism as an Off-Ramp

Erickson rejects a false choice between deterministic and agentic systems. Deterministic components ground an agent and free it to solve the uncertain, higher-order part of a problem.

LLo11yPop produced better eval results when retrieval agents used a wide, flat schema and simple SELECT, GROUP BY, and filtering operations rather than constructing complex joins. Access was read-only and exposed through an endpoint that could not write to the database. If the model repeatedly counted GPUs or joined data incorrectly, the team could provide a parameterized query pattern or runbook instead of asking it to rediscover the operation every time.

The same rule applies outside databases:

Delegate arithmetic and character counting to Python or a calculator.
Encode a known incident response in a runbook rather than reasoning from first principles during every incident.
Let an agent recommend a refund while a deterministic policy caps the amount and routes exceptions to a human.
Let an agent identify a potentially unhealthy GPU while a controlled tool performs any consequential reallocation.

Erickson's boundary is based on consequence. Discovery can tolerate candidate ideas that turn out to be wrong. Financial transactions, personnel changes, and GPU moves that may affect tens of millions of dollars of research capacity need predictable tools, strict governance, and human review. His formulation is not "never let an agent call a tool"; it is "do not make the consequential operation itself stochastic when deterministic software already fits it."

That boundary still requires adequate tool access. Erickson explicitly warns that insufficient tool access can be dangerous: the agent should discover and recommend within constraints, while a suitable tool performs the known operation. Reliability comes from assigning each part to the right mechanism, not from withholding tools indiscriminately.

Evaluate the Hierarchy as a Hierarchy

Erickson maps agent evaluation to a software test pyramid:

Component evals ask narrow questions of retrieval or worker agents, such as whether a query counts GPUs correctly. These are numerous and relatively cheap.
Integration evals test analyst agents that aggregate several workers. They require more orchestration and inference, so they cost more.
End-to-end evals test a supervisor's outcome against known evidence, such as analyzing 1,000 GPUs and comparing the result with ground truth. These are the fewest and most expensive.

The team used LLM-as-a-judge because semantically equivalent answers can have different wording. Erickson describes it as good enough for their use, not perfect. They also examined operational charts and sampled answers. The talk does not provide judge models, prompts, agreement rates, dataset sizes, or numerical LLo11yPop accuracy results.

Each additional agent can stack error on the output of the previous one. That makes hierarchy depth a reliability and cost trade-off, not an automatic good. The platform needs evals at each boundary so a top-level failure can be localized instead of attributed vaguely to "the model."

Erickson warns that an AI pilot at roughly 80% effectiveness may look impressive in a demonstration while remaining unacceptable for financial or similarly high-consequence work. He uses 80% illustratively; the presentation does not claim it as LLo11yPop's measured score or as a universal production threshold.

Match the Agent Shape to the Work

The presentation offers several agent archetypes. They are design vocabulary, not a claim that every platform needs each one:

A worker agent applies modest judgment at scale, for example inspecting 100,000 clusters for unusual power behavior.
A ruminative agent performs longer-running analysis of exceptions, perhaps comparing security, cost, and latency perspectives or using a chain of debate to propose a previously unnoticed failure pattern.
A middle-manager agent turns a bounded goal, such as reducing latency, into prioritized recommendations and tasks.
A consultant agent observes the wider system, spots recurring reasoning, and suggests where a deterministic operation could replace repeated agent work.
A tool-selector agent maps a requested operation to the appropriate tool and invocation. Some selected tools may be deterministic rather than model-based.
A director agent would run an Observe-Orient-Decide-Act loop over a constrained domain. Erickson presents largely autonomous director agents as future-facing, with recommendations and simulation preceding closed-loop control.

He also identifies practical starting points he calls dumb diamonds: tedious tasks that require a little judgment but consume disproportionate human effort. Examples include classifiers, content organization, scaled inspection, and procedural checks that humans currently approve with little scrutiny. For constraint navigation, Erickson points to an AlphaGo-like technique that chooses statistically promising moves in a search space rather than necessarily using an LLM; he says GPU arrangement and even reorganization problems can fit that category better.

Grounding, Identity, and Feedback Are Reliability Controls

In a long-running research loop, grounding frequency is a dial. Erickson uses NVIDIA's portable deep-research assistant, which he also calls AIQ, to illustrate a system that can reason with internal databases, repositories, and vetted documents while periodically revisiting its sources. Checking a database, document, or external source more often can improve factual alignment but slows the system and can complicate analysis. Grounding too rarely lets unsupported premises compound through extended reasoning. The correct interval depends on the task and should be evaluated rather than assumed.

Authorization should be narrower than the initiating user's authority. Erickson argues that running an agent as the querying person grants too much access. A purpose-built agent identity should be able to read only the page, database, or resource needed for its role, often with fewer rights than an intern. This turns specialization into a security boundary as well as a reasoning aid.

Finally, Erickson argues that agents should get better through use if teams build the right reinforcement and feedback loops; usage alone does not establish improvement. Those loops need measurement and human judgment about useful outcomes. People still supply taste, customer empathy, and engineering intuition. Teams should measure agent behavior and admit when an approach does not work instead of interpreting every output as evidence of progress.

Synthesis: A Practical Reliability Boundary

The following is synthesis from the presentation, not an architecture Erickson explicitly prescribed as a complete checklist.

For each proposed AI capability, separate the workflow into four zones:

Discover: Use a narrow agent where the problem benefits from semantic interpretation, anomaly discovery, or exploration of a large fuzzy space.
Ground: Give it vetted rare context, a constrained tool menu, and explicit query or runbook patterns. Record the evidence behind the recommendation.
Decide: Apply deterministic policy and human review in proportion to the consequence and reversibility of the decision.
Act: Execute through a least-privilege deterministic service, then capture the outcome as feedback without assuming that usage alone means success.

This framing gives teams a concrete answer to "where should the AI go?" Put it where uncertainty is valuable, and narrow or remove it where correctness depends on a repeatable operation. Evaluate each interface independently, because a reliable component cannot compensate for unchecked error accumulation across the whole chain.

Takeaways

Start with small, composable skills that do one thing well.
Supply company-specific rare context; do not mistake plausible zero-shot behavior for operational understanding.
Replace repeated model reasoning with deterministic queries, tools, and runbooks when the operation is already known.
Limit schemas, choices, permissions, and scope before reaching for a larger model.
Build component, integration, and end-to-end evals, accounting for stacked error and the rising cost of higher-level tests.
Keep humans and deterministic policy in the path of high-consequence actions.
Treat data quality, grounding cadence, identity, observability, and feedback design as first-class AI platform responsibilities.

Canonical source: Aaron Erickson, "Designing AI Platforms for Reliability: Tools for Certainty, Agents for Discovery", QCon San Francisco 2025, published by InfoQ on July 7, 2026.