Attributed is not incremental

In September I went through four abandonment flows for a pharmacy ecommerce brand. The analysis in front of me ranked them by revenue per recipient. Abandoned checkout came out well ahead, with a suggestion to make it the place to grow.

Here's the order rate for each, meaning the share of recipients (count) who placed an order, from June 1 to September 22:

• Abandoned checkout: 3.22%
• Abandoned cart: 2.21%
• Abandoned product (browse): 0.68%
• Abandoned site visit: 0.17%

Revenue per recipient fell in the same order, which is also the order of how close each group was to buying. Someone in the checkout flow had picked items and started to pay. Someone in the site visit flow had only been on the site. So the ranking measured intent, and it couldn't say how many of those orders the emails caused. No holdout had ever been run on any of the four.

Why the best flows look best

An email platform credits a message when someone opened or clicked it and then ordered inside the attribution window, whether or not the email made the difference. In one account I looked at, an open within five days was enough. That rule flatters abandonment emails most, because the people getting them already had something in a cart or had just been browsing, and some were coming back to finish anyway. The report can't tell you how many.

Tools also disagree. At an architectural products retailer, over four weeks this summer, the email platform credited email with roughly 7 to 12 times the revenue Google Analytics gave it on last click. The flows had only 165 clicks in that window, so most of their credit came from opens. The abandoned checkout flow was catching buyers already partway through a purchase, who finished through a direct visit or a branded search.

Meta ran the other way. Its own reporting claimed many times what the email platform gave it, because Meta clicks often lost their tracking tags. Each tool counts by its own rules, so two of them can claim one order.

A vendor up for renewal

In July the same retailer asked me for a neutral read on an abandonment-email vendor it was deciding whether to renew. The vendor identified anonymous site visitors the brand's own flows couldn't reach and sent them welcome and abandonment emails. The email platform credited those emails with about ten orders between May and early July.

With no holdout to go on, I went through the buyers one at a time. About 19 people bought after getting one of the vendor's emails:

• 11 were already known: four past customers and seven people on the email list.
• About seven were new, and their orders were small.
• None had gone quiet and come back. 18 of the 19 had been active on the site within a day.
• 11 bought without ever opening the vendor's email.

For those eleven, it's hard to argue the email did any work. Most of these orders were already in motion. The audience also left quickly: the vendor's emails had an unsubscribe rate of 2.31% (share of recipients, over 90 days), about three times the brand's usual 0.74%.

None of this is special to vendors. Any flow that fires after a recent visit or a filled cart is credited the same way, including your own.

How I read these reports

• I call it attributed revenue, in the column header and out loud, and I don't build a scaling case on it.
• I expect the ranking to follow intent, and when it does, I read it as a map of where buyers already were.
• I ask whether the flow has ever had a holdout.
• When a renewal or a budget depends on it, I look at who the buyers were and whether they opened the email at all.
• I put the email platform's number next to a second source, such as last-click revenue in Google Analytics. Neither is the truth, but the gap is worth knowing.

Cuts you can make now

The same logic makes cuts easy. If attributed revenue overstates what a flow causes, an email with very little attributed revenue is causing close to nothing, while its unsubscribes are real.

At the pharmacy brand, the third abandoned checkout email earned almost nothing in attributed revenue from 166 recipients and had an unsubscribe rate of 3.01% (share of recipients). The site visit flow was similar: very little from 1,192 recipients in about four months, with around nine unsubscribes. Both were safe to pause without a test.

Growing a flow is different. That checkout flow reached only about 22% of people who abandoned a checkout (unique people, June to September), because it only sends to people with email consent. More consent at checkout may be worth chasing, but until a holdout shows what the flow adds, I can't say what the extra reach is worth. Pushing harder for consent can also bring in subscribers who later mark the brand as spam.

Do this Monday: a holdout on one flow

Pick abandoned checkout, since that's where attribution flatters most.

1. Count how many people entered the flow in the last two weeks. A tenth of that is roughly your held-back group for a two-week test. If that's only a few dozen people, plan on a month or two instead.
2. Add a random split at the start of the flow: 90% of entrants (count) get the flow as usual, and 10% get nothing from it. In most email platforms this is a random split step at the start of the flow.
3. Keep offers, email copy and timing as they are while it runs.
4. At the end, pull orders for both groups from your store's own data over the same window. The email report won't show the held-back group, because they got no emails.
5. For each group, divide the people who ordered by the people in the group. The difference between the two order rates is what the flow adds. Multiply it by everyone who entered the flow and compare that with the orders the report credited.

The cost is small and capped. For two weeks, one person in ten misses the flow, so the most you give up is whatever it adds for that tenth. If it adds a lot, you have the number to justify growing it. If it adds little, you found out cheaply, and every later decision about that flow gets easier to price.

---

In one line: abandonment flows report the best numbers because they reach people who were already buying, so run a holdout before you put more money or reach behind one.