{"id":1723,"date":"2026-09-09T15:15:00","date_gmt":"2026-09-09T15:15:00","guid":{"rendered":"https:\/\/bizdom.app\/?p=1723"},"modified":"2026-09-09T15:15:00","modified_gmt":"2026-09-09T15:15:00","slug":"standing-holdout","status":"publish","type":"post","link":"https:\/\/bizdom.app\/en\/prestige-trailblazer\/standing-holdout\/","title":{"rendered":"Your revenue rose. Your control group would have told you why."},"content":{"rendered":"

A revenue increase after a launch proves nothing on its own, because a before-and-after comparison also captures whatever the market did during that window.<\/strong> DTC shipments fell 15% in volume and 6% in value in 2025 (Sovos ShipCompliant and WineBusiness Analytics, 2026), so the baseline itself is moving. A standing holdout \u2014 a random slice held back permanently and reported as a difference between groups \u2014 is what separates your work from the season.<\/p>\n\n

Take any result your team reported in the last year and try to state the counterfactual out loud: what would this number have been if we had done nothing? For nearly every DTC program in the category, there is no answer because no group received anything. The deck shows a line going up after a launch date, and the causal claim is carried entirely by the arrow.<\/p>\n\n

That was survivable when the channel was growing, because everyone’s arrows pointed the same way and nobody looked closely. It is not survivable now. DTC shipments fell 15% in volume and 6% in value in 2025, the worst year since the report series began, and the rise in average bottle price is a mix shift rather than premiumization (Sovos ShipCompliant and WineBusiness Analytics, DTC Wine Shipping Report 2026). When the ground under the baseline is moving, a before-and-after comparison reports the ground rather than the work.<\/p>\n\n

The mechanism that fixes this is old, cheap, and almost entirely absent from mid-tier DTC programs: a group that does not receive the thing you are measuring.<\/p>\n\n

The Standing Holdout<\/h2>\n\n

A holdout is a randomly selected slice of your list that is deliberately excluded from a program, held constant over time, and reported against. Three components make it work, and each is where most attempts break down.<\/p>\n\n

Component 1: Hold back permanently, not per campaign<\/h3>\n\n

Many teams already suppress a segment from individual sends and believe they have a control group. They do not. A different exclusion list each week measures nothing cumulative, because the same person gets three of your five campaigns, and the comparison collapses into noise.<\/p>\n\n

The standing version selects a slice once at random and holds it for a defined period: a quarter at minimum, a year if you can stand it. That group receives your baseline communication and none of the program you are evaluating. What you then get is not a campaign result but a cumulative one, which is the level at which the money actually shows up.<\/p>\n\n

Size it based on what you can afford to leave untouched rather than on a rule of thumb. A smaller holdout takes longer to say anything, and a larger one costs more in forgone contact, and both are legitimate positions depending on how contested your results are internally.<\/p>\n\n

Component 2: Assign at random, never by behavior<\/h3>\n\n

This is where good intentions destroy the result. The instinct is to hold back the members who seem least likely to buy because it feels like it minimizes costs. Do that, and your control group is made up of quiet members, your treated group is made up of active ones, and the difference you eventually report is a description of who you selected rather than what you did.<\/p>\n\n

Random assignment is the whole point. It makes the two groups alike in everything, including the things you never thought to record, so the difference between them at the end has one plausible cause left. That is a stronger claim than any dashboard can make, and it is available to a program of any size.<\/p>\n\n

One practical note: assign at the household or contact level and keep the assignment sticky. A member who lands in the holdout stays there for the period, including when a colleague wants to add them to a launch list because they are a good customer. That request will arrive, and holding the line on it is the job.<\/p>\n\n

Component 3: Report the difference, not the level<\/h3>\n\n

The output of a holdout is a difference between two groups over the same window, which changes what a good result looks like.<\/p>\n\n

In a growing market, level reporting flattered everyone: revenue rose, and the program took credit. In a contracting one, level reporting punishes everyone, including functioning work. A treated group that declined slightly, compared with a holdout that declined sharply, is a genuinely strong result, and in most decks in this industry right now, it is being reported as a decline.<\/p>\n\n

That reframing is the single most valuable artifact this system hands you. It lets you say something structurally true to ownership: the channel contracted, our program held a difference against it, and here is the group that shows what the year looks like without the work. This is also how you tell a real lift apart from a good quarter when the honest expectation is modest; personalization work has a documented revenue lift of roughly 5 to 15% (McKinsey), which is exactly the size of effect that a seasonal swing will hide.<\/p>\n\n

Report it as a rate rather than a total when the two groups are of different sizes, and present both side by side: revenue per contact in each group and the count in each group. Anyone senior enough to challenge the result will ask for the second number, and having it ready is the difference between a finding and an argument.<\/p>\n\n

What a Holdout Cannot Tell You<\/h2>\n\n

Being precise about the limits is what keeps this credible when somebody pushes back, so it’s worth stating two of them before anyone else does.<\/p>\n\n

A holdout measures the program you excluded, not the whole relationship. If your control group still receives your regular releases, your subscription shipments, and your tasting room, then what you are measuring is the marginal effect of one layer on top of everything else you do. That is usually the question you want answered, and it is not the same as knowing what your marketing is worth in total.<\/p>\n\n

A holdout also cannot tell you about effects that leak between the groups. Members talk to each other, forward emails, and see the same social posts, so a program with a strong word-of-mouth component will register smaller in a holdout comparison than it truly is. That is a reason to describe your measurement as conservative, which is a comfortable position to be in during a review, rather than a reason to skip it.<\/p>\n\n

The Objection You Will Hear Internally<\/h2>\n\n

Somebody will say that withholding your best program from a slice of paying customers is leaving money on the table, and in the narrow sense, they are correct. The holdout has a cost, and it should be named rather than hidden.<\/p>\n\n

Set it against what the alternative costs. Without a control group, every result you report is contestable by anyone in the room who prefers a different explanation, and in a down year, somebody always does. The holdout converts your entire program from an assertion into a measurement, and it does so permanently, across every initiative you run inside it. Priced against one quarter of unwinnable arguments about attribution, it is inexpensive.<\/p>\n\n

There is a second objection that arises later and matters more: somebody will want to release the holdout early because the results look good at week three. Early results move around a great deal before they settle, and a comparison stopped at the moment it looks most favorable is not a measurement of anything. Fixing the end date in advance, in writing, is what makes the eventual number worth presenting.<\/p>\n\n

The version that usually gets approved: start with a holdout on one program rather than on everything, run it for a quarter, and bring the difference to the next review. Nobody argues with the second one.<\/p>\n\n

This Month’s Action<\/h2>\n\n

Pick the single program you are most often asked to justify. Randomly select a slice of the eligible audience, tag it in your email automation platform, and exclude it from that program only for one quarter.<\/p>\n\n

Write down two things today, before any data exists: what you expect the difference to be, and what result would make you stop the program. Recording both in advance is what separates a measurement from a search for a flattering number, and it takes five minutes.<\/p>\n\n\n

\n
See how a standing holdout makes DTC lift defensible<\/a><\/div>\n<\/div>\n\n\n

P.S.<\/strong> The holdout has a second use that is worth more than the attribution argument. Once you have a group living without your programs, you can see what your baseline relationship actually is, which is the number that nobody in the category knows about their own list. Friday’s email is about what happens to all of this evidence afterward: why the answers a DTC team buys with a year of testing usually leave the building when the person who ran them does.<\/p>","protected":false},"excerpt":{"rendered":"

The mechanism that separates what you did from what the market did.<\/p>\n","protected":false},"author":1,"featured_media":1722,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[43],"tags":[],"class_list":["post-1723","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-prestige-trailblazer"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t