Attribution vs Incrementality: Are We Overstating CRM Performance?

This isn’t a dig at anyone; it’s an invitation for a proactive discussion and thought. Given what’s happening in the consumer market right now (price of fuel, interest rates, low consumer confidence,) many businesses are starting to “sweat their assets.” Whether it’s paid media, branding, out-of-home, or CRM, marketing teams are increasingly being asked to demonstrate ever increasing value. What’s working? How well is it working? And just as importantly what isn’t working, and where should resources be reallocated to drive better performance?

Lately, my Instagram feed has been flooded with ads from ambitious small agencies and individual operators making bold claims about performance improvements from “flows” and email programs. Before going any further, it’s worth acknowledging two things. Firstly, a lot of these operators are producing exceptionally high-quality work, emails that, in many cases, rival what you’d expect from much larger creative agencies. That in itself is impressive. Secondly, it’s not easy to step out and build something of your own. Starting an agency, even a small one takes a huge amount of effort, resilience, and risk. Anyone doing that deserves a level of respect. As Shopify has grown, so too have CRM tools that integrate with Shopify and are incredibly easy to use and often include powerful AI tools and reports. This has created a new type of “CRM practitioner” who probably has not been trained on the more traditional and wider fundamental aspects and principles of the industry.  When you look closely at how performance is being measured, most of the claims made by these agencies and individual operators rely heavily on email-attributed sales (or sms-attributed sales,) defined as the sales generated after a customer has opened or clicked a communication in a set amount of time. In theory, this makes total sense, but in reality it’s not giving you an accurate picture.

The challenge is that it’s not a true measure of CRM (email, SMS, etc) performance. Multi-channel attribution, by design, looks at which channels “touched” the sale. It doesn’t prove what actually caused the sale, which is ultimately what everyone is trying to understand. Attribution absolutely has a role, especially when viewed across the full channel mix. It helps answer the question; which channels are contributing to revenue at a higher organisational level. And yes, imperfect data is better than no data. But if you want to understand the actual impact of CRM specifically, multi-channel attribution models alone aren’t enough as these attribution models often fail to properly account for CRM in the purchase flow, this is due to the way attribution is generally calculated. That’s where incrementality comes in. Incrementality measures the difference your CRM activity makes, not just whether it was present. Unlike other channels, CRM boasts the benefit of being able to include a robust holdout (or control group), this allows for a different and more accurate form of measurement, incrementality. Incrementality measures uplift above a like audience who did not receive a particular communication. For those who are not familiar with this concept, you split your audience into two groups so that you are somewhat measuring like-for-like (apples with apples); one receives the communication (target), the other doesn’t (control). You then compare the difference in behaviour between the two. That difference (the uplift) is what you can confidently “bank” as the direct impact of your email, SMS, or push campaign. It’s the same principle used in drug trials. One group gets the treatment, the other a placebo. Success isn’t measured by outcomes alone; it’s measured by the difference between the two groups. When applied to CRM, this approach gives you a far more accurate view of performance. Yes, the numbers might not look as impressive. But they’re real, measurable and far more manageable. And importantly, incrementality helps strip out “noise.” Performance is influenced by far more than just your email program or your CRM strategy; it’s influenced by such factors (“noise”) as seasonality, pricing strategies, product assortment, stock availability, paid media investment, and broader economic conditions all play a role. A customer might click an email and purchase, but was it the email? Or was it the new product range? Or a new 15% promotion that you didn’t run last year? Attribution alone can’t answer that. There’s also the “first-time lift” effect. When you introduce a new campaign, flow, or offer, you’ll often see a spike in performance simply because it’s new. But to truly understand its value, it needs to be tested repeatedly over time, not judged on a single result.

Some CRM platforms and strategies are clearly built for optimisation (A/B testing) rather than causation (which alone can have many added benefits.) And that distinction matters. Because what I’m seeing more is smaller operators leaning heavily on pure email attribution-based results to promote their services. It looks great from a marketing perspective, big numbers, strong claims to get that “free audit.” But it’s not a true reflection of what CRM is actually contributing. This might be a good short-term strategy (a “warm and fuzzy to get some buzz around email marketing) but when times get tough (and they most certainly will,) you can’t rely purely on email attribution figures when looking at true impact and where to further invest resources.

If a business ramps up media investment and a customer also clicks an email, both channels are influencing the outcome. Attribution spreads credit across touchpoints, but if you’re isolating email performance, only incrementality can tell you what email did. That’s exactly why measurement matters. Without the right reporting mix, you risk mistaking correlation (“both CRM and paid media were present around the sale”) for causation (“would the customer have behaved differently if CRM didn’t exist?”) and that’s where the real issue lies. So I’m curious, especially from those who currently push email or sms attribution rather than uplift (and who may not be familiar with this measurement concept) are your clients asking about true incremental impact of various initiatives? Or are they satisfied with attribution-based reporting? Are you finding your clients are putting increasing pressure on you to demonstrate how in particular various initiatives are supporting the business? Because the answer to that question says a lot about where CRM is heading and how seriously it’s being treated as a commercial growth lever and how some smaller businesses are defining success.