# The data is right. The conclusions are wrong.

**Written by JD Prater — 3x VP of Marketing.** This piece is part of *Open Folders*, which sits at the intersection of marketing science, consulting research, and practitioner experience, opening the parts of the discipline most marketers never got taught, so they can lead the whole of it, not just the slice they came up in.
Newsletter: https://jdprater.substack.com
Website: https://jdprater.com
Original post: https://jdprater.substack.com/p/the-data-is-right-the-conclusions

Most performance dashboards are accurate, and somehow that's the most dangerous part.

The data is clean, the reporting is real-time, the ROAS numbers move when you pull the levers. Everything looks like it's working. And then, gradually, it isn't. CAC climbs, efficiency fades, the levers stop responding the way they used to. The team tries harder. The numbers get worse.

I've been in that room. It's a confusing place to be, because nothing looks broken. The dashboards are green. The process is sound. The people are good. And yet.

This is the death spiral I wrote about last time. What I didn't cover is why it feels rational at every step. Why smart teams make this mistake repeatedly, with good data, and don't see it coming.

The answer is a measurement problem. Not a people problem. The people are usually fine.

## You're measuring the tide

Dr. Grace Kite found something important in her data. Return on ad spend correlates strongly with category demand. When demand is high, ROAS goes up. When it softens, ROAS falls. The weather, the economy, competitor moves, things entirely outside your control, are moving your dashboard more than your marketing decisions are.

Your team is a boat floating on a tide. Don't be too quick to call them geniuses when results are good, or fire them when results fall. The tide was doing most of the work. It just didn't show up in the debrief that way.

The deeper problem is that teams react to the signal constantly. ROAS moves, the dashboard flags it, decisions change. Each one looks right in isolation. Away from the dashboards, more sales go through on deal, campaigns become dizzyingly changeable, and the big picture quietly deteriorates. Spend climbs faster than sales. Margins fall.

I've made versions of this mistake myself. When a category is growing, acquisition looks cheap and easy. Brand gets cut because the performance numbers are strong and brand is harder to justify. That's not irrational behavior. More like rational decision-making with an incomplete picture.

Then the category cools. I'm looking at you, home fitness equipment, meal kits, at-home everything. Products that looked like marketing miracles in 2020 and quiet disasters by 2023. Some brands used that window to build something durable. Most of us optimized into it and emerged with no buffer when demand normalized. ROAS dropped. CAC climbed. The instinct, backed by the data, was to optimize harder, cut what couldn't prove its return. Brand went first.

The brands that came out stronger weren't just luckier. They made a genuine choice to invest through the peak rather than optimize into it. Every decision in the spiral is defensible. None of them constitute a strategy.

The measurement system never flagged any of it. It wasn't designed to, and frankly nobody had asked it to.

## Built for a different job

Attribution works exactly as designed, which is sort of where the trouble starts.

It was built to capture demand that already exists: someone searches, clicks, converts. It's precise, fast, and genuinely useful for what it does. I'm not here to bury it. I use it. Most of us do.

What it can't see is the work that makes your brand the one someone thinks of when they're finally ready to buy. That work produces memory, not clicks. And it takes months to accumulate and months to erode, neither of which registers in a standard measurement window.

So the performance ads look like they're doing all the work. The brand ads look inefficient. Budgets follow the signal.

This is where the reliability trap kicks in. Rarely do marketing teams choose attribution because it's the most predictive measure. Rather, they choose it because it's the most defensible one. Same inputs, same outputs, every time. It survives a CFO review. I know because I've been in that review. I've picked the number I could defend, not the one I trusted most. MMM requires judgment, interpretation, a willingness to act on a model rather than a dashboard. So we default to what's reliable over what's valid, and call it rigor.

This is, to put it mildly, a problem.

It's getting harder to ignore. As AI search reshapes how people find brands, fewer journeys produce the clicks attribution depends on. Someone asks Claude for a recommendation and gets your brand name back. No click. No record. Memories still being encoded, brand associations still being formed, invisible to the dashboard that's supposedly tracking everything.

Here's what compounds it. Research across more than 1,000 brands finds that 60% of business growth comes from factors that never touch a marketing dashboard: availability, distribution, macroeconomics, seasonality, competitor moves. We're optimizing 40% of the equation while the other 60% runs unaccounted for. (Source: Analytic Partners ROI Genome, *Growth Beyond Marketing*, 2025)

Which means the metrics can glow while the business quietly bleeds. And when that gap shows up in the numbers, marketing doesn't just make bad decisions. It loses the argument. Budget gets cut not because the marketing failed, but because nobody could explain the gap.

## What the model sees that the dashboard doesn't

Marketing mix modeling sits alongside performance measurement, giving you the context attribution can't provide on its own.

The first time I saw a clean MMM output, it was uncomfortable. A well-run model separates base sales — what you'd generate without any paid media — from what your campaigns actually drove. When base sales are rising, your category tailwind is taking credit your campaigns don't deserve. When they're falling, no amount of optimization fixes that. The model just makes it visible. Which is uncomfortable, and also the point.

Note: base sales represent your structural demand floor — shaped by brand equity, distribution, pricing, and macro conditions — not simply "sales without paid media." Rising base sales may reflect brand investment compounding over time, not just category tailwind. Both look identical to attribution. (Source: Mutinex 2026, Meta/DLM study)

MMM also shows you where diminishing returns are already biting before your dashboard admits it. That's the conversation I'd rather have early, when there's still room to move, than late, when the options are all unpleasant.

We use Mutinex for this work. What it surfaces isn't visible in any attribution model I've used. Attribution handles its job well. MMM handles a different one.

## The bill arrives later

Rarely is the death spiral the result of careless teams. More often it happens because a system that could only ever see captured demand produced confident, rational-looking signals, while the depletion underneath stayed invisible.

By the time it shows in the numbers, you're already well into it.

Rebuilding from a depleted position takes years and costs substantially more than maintaining the asset would have.

## The part where I tell you it's complicated

MMM isn't a perfect solution. It's retrospective, not real-time. The data requirements are real. And the discipline required to actually govern by the outputs — rather than relitigate the model every planning cycle — is harder to build than the model itself. Much harder, in my experience.

Building the model is easier than getting an organization to actually be governed by it. That's where most of the time and frustration goes.

If I'm being fully honest: measurement problems are often strategy problems in disguise. If you never made an explicit choice about what you're building — brand as a long-run asset rather than a cost center — you'll measure whatever's easiest to defend. Attribution fills that vacuum perfectly. Nine times out of ten, the tool isn't the problem. The choices made around it are.

But once you've made the choice, running budget allocation off attribution alone is choosing to stay blind to the things most likely to determine long-run performance.

Your dashboard is almost certainly accurate. Whether it's measuring the right things to make durable decisions is a different question entirely.

Most of the time, it isn't.

*Still thinking, JD*