A lot of engineering leadership today reads like an AI press release: every team is shipping faster, every metric is up and to the right, every pilot is a success. In this InfoQ presentation, Justin Reock pushes back against that narrative — not from a place of skepticism, but from a place of measurement. His argument is that AI-assisted engineering is real, valuable, and worth pursuing, but only if you instrument it honestly and structure your organization to absorb it. Otherwise you end up celebrating activity while the underlying business outcomes drift sideways.
Beyond the Hype: Why a Framework Matters
Reock opens with a familiar problem: most leaders cannot tell whether their AI investments are actually paying off. Vendors quote eye-popping productivity gains; engineers quote anecdotes; finance asks for ROI. Without a framework that ties tool usage back to outcomes, the conversation devolves into vibes.
The talk recommends two well-known frameworks as a starting point:
- SPACE — measures Satisfaction, Performance, Activity, Communication, and Efficiency. Crucially, it is multi-dimensional, so a gain in one dimension doesn't compensate for a regression in another.
- Core 4 — a more focused set of operational metrics for delivery throughput and quality.
The reason for using both is to avoid the trap of optimizing one slice of engineering at the expense of another. AI tools tend to move "Activity" and "Speed" first; SPACE forces you to also look at "Satisfaction" and "Communication," which is exactly where AI adoption tends to quietly hurt teams.
The AI Learning Curve and the GenAI Divide
A core piece of research Reock cites is the GenAI Divide: roughly 95% of generative AI pilots fail to deliver value beyond an initial prototype. Most teams get a sugar high from their first month — autocomplete is fun, code generation feels magical — and then plateau or regress as integration friction, review overhead, and quality issues catch up.
The implication for leaders is sobering: linear extrapolation from a pilot is almost always wrong. The honest curve looks more like:
- Initial boost — engineers move faster on familiar tasks.
- Friction trough — review burden climbs, debugging unfamiliar AI-written code gets harder, ownership blurs.
- Sustainable plateau (or not) — only teams that reorganize around AI continue to see compounding gains.
Plan for the trough. Budget for it.
The Cobra Effect: When Metrics Bite Back
The presentation's most quotable warning is the "Cobra Effect" — named after the colonial-era anecdote in which a bounty on cobras led to people farming cobras for the bounty. When the program was canceled, the farmed cobras were released, and the cobra population was worse than before.
In engineering, the same dynamic shows up everywhere AI is measured by superficial output:
- Reward "lines of code shipped"? You get more lines, more bugs, and more tech debt.
- Reward "PRs opened"? You get small, low-value PRs and review fatigue.
- Reward "AI tool usage rate"? You get engineers running the tool to satisfy the metric.
Reock shares a case study where an organization measured AI-generated lines of code and saw output skyrocket — alongside a meaningful drop in maintainability. The fix was to pivot to value-aligned outcome metrics: customer impact, change failure rate, satisfaction scores.
The principle is simple but uncomfortable: if the metric is easy to game, AI will help your team game it.
Augmenting Throughput Without Replacing Judgment
Reock is bullish on AI but unequivocal that it should be deployed as a co-pilot, not an autopilot. Concretely, that means:
- Keep humans in every meaningful review loop.
- Use AI to remove drudgery — boilerplate, test scaffolding, doc updates, refactor sweeps — so engineers spend more time on the parts that need judgment.
- Embed AI across the SDLC, not just at code generation. Specs, design docs, code review, test generation, incident summaries, and release notes are all higher-leverage than another autocomplete plugin.
Practical examples discussed in the talk include:
- Automated test generation — yields faster releases and better coverage, but only when paired with rigorous review.
- Code review acceleration — AI flags candidates; senior engineers focus on architecture and risk.
- Documentation generation — NLP tools draft API docs, humans validate and extend.
The thread running through all of these is augmentation: the AI proposes; the engineer disposes.
Leadership Lessons
The leadership message is the heart of the talk. Reock argues that successful AI adoption is mostly a culture problem, not a tooling problem.
- Champion data-driven decisions. Insist on hard metrics before scaling a tool, and on continued measurement after.
- Create psychological safety around AI experiments. Engineers will not honestly report what is and isn't working if failure is punished. Reduce "developer fear" so teams will surface problems early.
- Patience over theatrics. Iterative improvement beats a flagship rollout. Expect months of tuning before pilots become platforms.
- Invest in tight feedback loops. Shorten the distance between "we tried this AI workflow" and "here's the impact on outcomes." Without that, you cannot tell augmentation from decoration.
Final Takeaways
If there is one thing to carry away from Reock's talk, it is that AI-assisted engineering is a leadership discipline, not a procurement decision. Your tooling choices matter, but they matter far less than:
- Measuring the right things (SPACE + Core 4, not just lines of code).
- Anticipating the GenAI Divide and planning for the friction trough.
- Avoiding the Cobra Effect by choosing outcome-aligned incentives.
- Treating AI as a force multiplier for human engineers — not a replacement for them.
Done well, AI compounds the strengths of a healthy engineering culture. Done badly, it amplifies its dysfunctions just as effectively. The leaders who will win are the ones who invest equally in both halves of the equation.
Reference: Leadership in AI-Assisted Engineering