Incrementality testing without a data science team

You can measure incrementality without a data science team if you run one well-designed holdout at a time, decide the success rule before it starts, and use the platform lift tools or a simple geo split instead of building models. By the end of this post you will know which test design fits which channel, what each one needs, and where marketing mix modeling comes in.

Attribution counts conversions near an ad, not conversions caused by it

Attribution answers "which touchpoints came before this conversion". Incrementality answers "how many conversions would not have happened without this spend". The two numbers diverge most in the channels that look best in a report.

The mechanism is selection. Retargeting, branded search and prospecting audiences built on lookalikes all reach people who were already likely to buy. The ad sits in their path, so every attribution model gives it credit. Whether you use last click, data-driven attribution or multi-touch attribution, the model distributes credit among touchpoints that happened. It never sees the purchase that would have happened anyway.

For example, a retargeting campaign reports 500 conversions in a month. A holdout test on the same campaign finds that the group that saw no ads converted almost as often, and only 150 of those 500 were incremental. The platform cost per conversion and the cost per incremental conversion now differ by a factor of three. These figures are an illustration, not a benchmark.

The only way to see the conversions that would have happened anyway is a comparison group: people or regions that could have seen the ad and did not. That is all an incrementality test is. Jim Gianoglio makes the same distinction between assigning credit and measuring incrementality in the interview summarized here, and it is the core of the question.

How to run incrementality tests without a data science team

The work is mostly design and discipline, not statistics. The platforms and open-source tools do the estimation.

1. Pick one decision, not one channel. Write the decision the test will inform: "cut retargeting by half", "keep branded search", "expand Meta prospecting to a new country". The head of performance owns this. The artifact is one sentence with the decision and the metric that settles it, such as purchases or qualified leads.

2. Choose the design that fits the channel. There are three practical options, compared in the table below. Use the platform lift tool when the spend is concentrated in one platform. Use a geo experiment when you need to test a channel that cannot hold out individual users, or to compare across platforms. Use an audience holdout in your own channels, such as email or CRM journeys, where you control who gets the message.

3. Check that the test can detect an effect. A test that is too small will come back "inconclusive", which is the most common way these projects die. Platform tools show a power or feasibility estimate before launch. For geo tests, GeoLift, an open-source R package published by Meta Open Source, includes power calculations and market selection. If the estimate is weak, extend the duration or increase the holdout before launching, not after.

4. Write the rules before launch. The artifact is a one-page test plan: the decision, the metric, the design, the holdout size, the start and end dates, and the result that would change the decision. Share it with the CMO and finance before the test starts. This stops anyone from reading the result halfway through and calling it.

5. Freeze the campaign while the test runs. No budget changes, new creatives or audience changes in the tested campaigns, and no overlapping tests on the same audience. Any change mid-test mixes two effects into one number.

6. Read three numbers, then act. The incremental conversions, the cost per incremental conversion, and the confidence or credible interval. Compare the cost per incremental conversion with the platform-reported cost per conversion. The ratio between them is the correction you apply to that channel's attributed numbers until the next test.

Three designs you can run without a model of your own

What the platform tools require

Google Ads describes Conversion Lift based on users with eligibility requirements, as of October 2026: at least 1,000 observed conversions, a study budget of at least USD 5,000, and a compatible conversion action. The help page lists Display, Search, Video, Demand Gen, App and Performance Max campaigns, and says the tool is not available to every account. It recommends at least 14 days of study time. There is also a geography-based version that needs campaigns targeting a single country, and Google lists it as a beta.

Meta offers conversion lift and holdout tests through its Experiments and Test and Learn tools, with a randomized control group that does not see the ads. The control group size, the minimum number of events and whether the test needs Conversions API are set by Meta and change over time; check them in Meta's help on holdout tests before you budget the test.

Both platforms measure their own ads with their own data. That makes the tools useful and free, and it is also the reason to run at least one geo test that you design and read yourself.

Where marketing mix modeling fits

Experiments answer one question at a time, for one period. Marketing mix modeling (MMM) answers a different one: how all channels together contributed over a longer period, including those you cannot test easily, such as TV or out-of-home. The introduction to marketing mix modeling covers the method itself.

The two work best together. Experiments give you causal results for a few channels, and an MMM uses those results to anchor the channels it cannot test. Both major open-source MMMs are built for this. Google's Meridian is a Bayesian model in Python that, as of October 2026, includes methods to calibrate the model with experiments. Meta's Robyn, still labeled experimental, describes calibration with geo tests and lift studies.

The difference for a team without data scientists is effort. A lift test can run with the people you have. An MMM needs two or more years of consistent spend and outcome data, someone who can judge the model's assumptions, and a refresh cycle. If you do not have that person, run experiments first and buy or commission the MMM later.

How you know a test result is trustworthy

Most failed tests fail in the design, not in the math. These are the checks to run before you act on a result.

• The test reached its planned duration and size. If it was stopped early because the result looked good, treat it as directional.
• The confidence matches the decision. Google Ads, for example, shows study power and treats results below its recommended level as directional. A directional result can justify another test, not a budget cut.
• Nothing else changed. Check for promotions, stock-outs, pricing changes or a competitor event in the test period, and check that the holdout was not reached by another campaign aimed at the same people.
• The conversion lag is covered. If customers take three weeks to buy, a two-week test misses most of the effect.
• The tracking held. If the conversion event broke during the test, the result measures the break. Check event volume across the test window.

For a CMO, the question to ask the performance team is: "Which of our channels has an incrementality test from the last 12 months, and what did it change?" A team that tests answers with a channel, a date and a decision.

When this does not apply

• Low conversion volume. If you have a few dozen conversions a month, most tests will not reach a usable confidence. Test on a higher-volume proxy, such as qualified leads, or run fewer, longer tests.
• Long B2B sales cycles. When the sale closes months after the click, test on an earlier stage you trust, and say so in the result.
• Channels you cannot switch off. Brand-level TV or sponsorships often cannot be held out cleanly. That is where MMM, not experiments, does the work.

What to check this week

• List your three largest channels and write down the last time each one had an incrementality test. Pick the one with the most spend and no test.
• Check whether that channel's account meets the platform's lift requirements, or whether you have conversions by region for a geo test.
• Write the one-page test plan, with the decision and the success rule, and share it before anything is launched.