The Manager's New Brief

15.1

Ch. 15 — Measuring What Matters When Productivity Is Easy to Fake

Part IV — Management in the AI era

💡 Core idea

When AI can generate almost unlimited activity — text, tickets, code, slides — activity is no longer a safe proxy for value. Managers must measure outcomes, quality, and health, or they will reward noise.

If a metric can be inflated by generating more artifacts, it is no longer a safe measure of performance. That is the whole chapter, in one sentence. The rest is how to replace the metric without pretending to have a perfect dashboard.

Activity breaks first

Bharat Sharma argues that activity metrics such as lines of code or pull-request counts are the wrong unit once generation is cheap, and that you should measure system outcomes. The underlying studies do conflict, and the conflict is useful rather than a reason to stay vague. Noy and Zhang (2023) found that ChatGPT cut time on midlevel professional writing tasks by 40% and raised quality by 18%. METR (2025), in a randomized trial with 16 experienced open-source developers working 246 real issues, found that allowing early-2025 AI tools increased completion time by 19% — even though the developers expected a 24% speedup going in, and still believed they had been 20% faster after the study. METR later said those results no longer reflect current tool impact, and a February 2026 follow-up had design problems, so do not treat the 19% slowdown as a current law. The durable lesson is the perception gap: do not take self-reported speedup as productivity, and do not assume one setting (writing tasks, customer support, mature codebases) generalizes to another.

Karim Jedda writes about engineering management after the cost of writing code collapsed. The useful generalization is broader than engineering: when first drafts are cheap, the management system has to change with them — including what you count.

Speed is not productivity

James Shore, measuring AI’s impact on delivery speed, argues that delivery speed is not productivity, which he frames as value produced relative to cost (money spent). Read the piece before you adopt his framing as a formula; the distinction is already usable. A team can move work through a board faster and still destroy value.

Shore also argues that if the tool does not reduce maintenance cost, “more output” may be a false win. ordep.dev puts the bottleneck where it usually was: understanding, reviewing, and changing a system safely — not writing the first draft. The same pattern appears in analysis, support, and content: generation is not the hard part.

Bjorg (2026), in “The AI productivity gap,” walks through a worked example, not a study: if a senior’s coding time falls from 1.5 hours to 0.5 hours at a 3× speedup, the whole day only shrinks from 8 hours to 6.75 — “about 15%.” A junior in the same arithmetic saves more of the day (“about 25%”). The author flags that the numbers are arguable. Use the piece as a reminder to do the arithmetic for your mix of work, not as a measured productivity law, and do not multiply a local speedup by a full-time-equivalent and call it productivity.

Aviv Ben-Yosef again: do not manage the token invoice and miss the process waste.

Volume is easy to misread as productivity. Anthropic reports a rise from around 2,500 merges a day in May 2026 to more than 30,000 in September, on its own monorepo, and says the constraint became human-scale infrastructure. That is a self-report from an unusual workload; do not copy the number as a target.

Token spend can also hide waste. Dimitri Mazmanov, writing about his own Claude Code setup at Spotify, found that much of the agent’s work was I/O, not thinking, and measured mean bulk-read savings of “around a whopping 90%” on tests in a Java monorepo by routing large reads to a cheaper model. This is one engineer’s setup and test, not an audited company-wide saving.

The scarcer measurement problem is the quality bar itself. Hiten Shah writes: “AI is forcing us to make more of our judgment reusable. That’s the part of evals I think people are underestimating.” In the accompanying essay, “A useful eval makes some of that judgment explicit enough to reuse.” Sampled review against a written bar belongs on the scoreboard, not only a count of output.

Atlassian’s AI Collaboration Index (Chapter 12) is a reminder of the measurement trap at company scale: workers “say AI makes them an average of 33% more productive,” while “96% of companies have not seen dramatic improvements in organizational efficiency, innovation, or work quality.” Self-reported speed is not the same as a change in the work. Winter-Tear’s pitfall is the other half: agents inherit the operating model you already have, including weak measures.

Pair outcomes with quality and health

For each major goal, keep three kinds of measure:

Itamar Gilad’s competence/judgment gap belongs on the scoreboard (Chapter 6): sampled quality and the ability to explain the work, not only volume. If output doubles and rework, defects, or confusion rise, you are not more productive.

Discord’s engineering blog argues for fewer, better metrics rather than a dashboard of everything. A short team scoreboard, visible and revisable, is enough. James Shore has also tried to describe a useful productivity measure in practice; read it before adopting any formula as yours.

Mike Fisher, drawing on James C. Scott, warns that “The danger arrives on the day we start managing the company to make it match the dashboard.” “Treat every metric as a compression, and ask what got compressed away.” The first rotation can look like a success while it spends stored, invisible capital — the informal work a scoreboard does not show. Once that knowledge is gone, it is hard to put back.

A regular, small sample of real output scored against a quality bar will tell you more than a count of artifacts. Expect gaming. Design for it.

Do not rank people on tool activity. Used clumsily, “AI productivity” scores punish careful work, encourage hidden tool use, and damage trust (Chapter 11). If you look at tool use at all, use it to find coaching needs and workflow friction, not to build a league table.

Tools

🤖 AI-era tip: If a model can game the metric, retire the metric

If a measure can be inflated by generating more text, tickets, or code, it is no longer a safe measure of performance. Shift evaluation toward outcomes, sampled quality, and stakeholder value. Be equally careful with individual “AI productivity” scores. If you measure tool use at all, use it to find coaching needs and friction, not to rank people.

Takeaways

Evidence