Measurement Myth Buster quiz.
Explore experiments, attribution and the limits of marketing evidence. Ten questions drawn at random from 25, an answer explained after each one, and a reward at the end: the experiment design pocket card.
Pick one answer, or two where it says so. After each question you see the answer and why. No timer, no sign-up. Whatever you score, you take away the experiment design pocket card.
Questions you have seen are remembered on this device for 30 days so new rounds feel fresh. Nothing is sent to us.
Learn first, play later.
Every question in this level's bank, with the correct answer and a short explanation. A round draws ten of these at random and shuffles the options. History questions link to the brand's own archive or the credited agency; scenarios are teaching examples, not client stories.
A brand randomly assigns eligible people to an ad test and a holdout. What is the holdout for?
- Ensuring every unexposed person has no prior brand awareness
- Guaranteeing that exposed and unexposed outcomes will match
- Allocating credit to the final touchpoint
- Estimating what might happen without the ad treatment (correct)
Answer: Estimating what might happen without the ad treatment. A comparable untreated group helps estimate a counterfactual. Randomisation improves comparability but implementation still matters.
A brand’s attribution dashboard credits the last click. What does that establish?
- That all earlier touchpoints had zero effect
- The isolated causal effect of the final click
- That the customer would not buy without that click
- Credit under a particular allocation rule (correct)
Answer: Credit under a particular allocation rule. Attribution assigns credit using rules or models. It does not automatically show what would happen without the marketing.
A brand test shows a small lift with a very wide confidence interval. What should the team recognise?
- A small positive estimate proves a commercially useful effect
- The point estimate is certain despite the wide interval
- The most optimistic bound is the best forecast
- The estimate is uncertain (correct)
Answer: The estimate is uncertain. A wide interval means the data permit a broad range of effects under the analysis. Report uncertainty with the estimate.
Which TWO strengthen a brand experiment’s analysis plan?
- Stop whenever the chart looks exciting
- Define the primary outcome in advance (correct)
- Hide all non-significant results
- Specify a valid stopping rule (correct)
Answer: Define the primary outcome in advance + Specify a valid stopping rule. A defined outcome and stopping rule reduce avoidable analytical discretion. Transparent reporting remains necessary.
A brand stops a test the first time its p-value crosses a threshold. What is the concern?
- Every threshold crossing has the same meaning under repeated checks
- The stopping rule cannot affect interpretation
- Unplanned repeated peeking can distort error rates (correct)
- The earliest positive result is the least biased estimate
Answer: Unplanned repeated peeking can distort error rates. Choose an appropriate stopping rule in advance or use a valid sequential method. Repeated informal peeking can mislead.
A brand compares a test city with a control city during a local festival only in the test city. What is threatened?
- The festival is irrelevant if the cities have similar populations
- Any city can be a valid control without comparison checks
- Comparability of the counterfactual (correct)
- The test city’s entire growth can be assigned to media
Answer: Comparability of the counterfactual. A different local event can affect outcomes independently of the campaign. Address that difference in design or analysis.
A brand sees both ad spend and sales rise during holidays. What prevents a simple causal reading?
- That seasonality proves advertising had no effect
- That two rising series establish an experimental result
- Seasonal demand may influence both (correct)
- That advertising necessarily caused the whole sales increase
Answer: Seasonal demand may influence both. A shared driver can move spending and sales together. The observed correlation does not isolate advertising’s effect.
A brand’s awareness is 30% in test and 25% in control. What is absolute brand lift?
- 55 percentage points
- 5 percentage points (correct)
- 20 percentage points
- 1.2 percentage points
Answer: 5 percentage points. 30% - 25% = 5 percentage points. Relative lift is 5 ÷ 25 = 20%, a different expression.
A brand’s target-market survey reports prompted recognition. What question would measure unprompted recall instead?
- Have you heard of our named brand?
- Do you recognise this displayed logo?
- Which brands come to mind in this category? (correct)
- Which name in this list looks familiar?
Answer: Which brands come to mind in this category?. Unprompted recall asks people to retrieve names without supplying them. Prompted recognition tests a different memory task.
A brand’s customer survey excludes people who stopped buying. What is missing for churn diagnosis?
- Evidence from former customers (correct)
- Treat the active-customer list as the complete former-customer population
- Infer churn reasons from repeat buyers’ satisfaction alone
- Assume loyal customers explain all reasons for leaving
Answer: Evidence from former customers. Former customers can reveal reasons that retained customers cannot represent. Sampling should fit the question being investigated.
A brand uses a before-and-after sales chart with no comparison group. What is the cautious description?
- Evidence that every new sale was caused by the campaign
- A definitive estimate of the campaign’s isolated effect
- An observed change over time (correct)
- An equivalent substitute for random assignment
Answer: An observed change over time. A time comparison can be informative, but it does not by itself exclude seasonality, distribution or other changes.
Which TWO statements about brand attribution are accurate?
- It can differ across reporting systems (correct)
- It always proves causality
- It makes holdout tests unnecessary
- It assigns credit under a rule or model (correct)
Answer: It can differ across reporting systems + It assigns credit under a rule or model. Attribution depends on rules and data. Different systems can disagree, and causal questions still require appropriate evidence.
A brand optimises every decision to clicks. What is a likely measurement blind spot?
- Every higher click total indicates stronger customer fit
- Clicks and profitable demand are necessarily the same
- Whether those clicks create useful business outcomes (correct)
- A click objective measures all downstream effects
Answer: Whether those clicks create useful business outcomes. A narrow objective can improve while broader outcomes worsen. Use guardrails aligned with the business goal.
A brand wants to report campaign ROI, not revenue ROAS. Which extra information is essential?
- Only the platform’s revenue ROAS label
- Relevant costs and an appropriate profit or contribution measure (correct)
- Only the attributed revenue total
- Only impressions and reach
Answer: Relevant costs and an appropriate profit or contribution measure. ROI needs a defined return and investment basis. Attributed revenue divided by ad spend is a different measure.
A brand’s experiment has people sharing the ad with the holdout group. What is the concern?
- The holdout is guaranteed to remain untreated
- Sharing makes randomisation unnecessary
- Treatment spillover can blur the comparison (correct)
- Spillover cannot affect the estimate
Answer: Treatment spillover can blur the comparison. Spillover can reduce separation between groups. Assess its scale and what it means for the effect being estimated.
A brand uses a survey sample of existing fans to estimate national awareness. What is the main limitation?
- National awareness equals existing-fan recognition
- The sample does not represent the target population (correct)
- Large sample size automatically repairs selection bias
- Brand fans can stand in for all potential buyers
Answer: The sample does not represent the target population. Sample size cannot repair a systematically unrepresentative sampling frame. Define the population first.
A brand’s marketing mix model attributes sales to media. What should stakeholders remember?
- A single model should never be checked against experiments
- Model assumptions and uncertainty affect the estimates (correct)
- The fitted model removes all omitted-variable concerns
- Every coefficient directly observes a causal mechanism
Answer: Model assumptions and uncertainty affect the estimates. Models require assumptions and validation. Interpret estimates alongside uncertainty and other evidence.
A brand measures search interest after a campaign. What should the team avoid assuming?
- All changes in search were caused by the campaign (correct)
- A campaign-time trend is equivalent to random assignment
- Every search change was caused by the campaign
- A search trend proves all searchers became customers
Answer: All changes in search were caused by the campaign. Search can be a useful signal, but news, seasonality and competitors can also change it.
A brand tests two headlines but changes the price only in one version. What can the result isolate?
- The price’s effect in every market
- The combined package of headline and price (correct)
- The logo’s causal effect
- The headline’s isolated effect
Answer: The combined package of headline and price. The versions differ in two dimensions. Separate changes or use a suitable design if the individual effects matter.
A brand’s two equal-size test groups produce 110 and 100 sales. What else is needed beyond the 10-sale difference?
- Any positive difference is commercially decisive
- Equal group sizes eliminate all uncertainty
- The difference alone proves a stable positive effect
- An uncertainty assessment and checks on test validity (correct)
Answer: An uncertainty assessment and checks on test validity. The observed difference is not enough to judge reliability. Sample variation and implementation can affect interpretation.
A brand’s sales data contain repeat orders and duplicated transaction IDs. What should precede analysis?
- Assume transaction IDs can never be duplicated
- Delete all repeat purchases regardless of the question
- Define the unit and deduplicate appropriately (correct)
- Count each data row as a new customer
Answer: Define the unit and deduplicate appropriately. A transaction, order and customer are different units. Clean data according to the metric you intend to measure.
Which TWO make a brand CAC comparison interpretable?
- Comparable cost scope and timing (correct)
- Mix renewals into one denominator only
- Consistent new-customer definitions (correct)
- Use media spend in one market and all costs in another without labels
Answer: Comparable cost scope and timing + Consistent new-customer definitions. Consistent definitions and scope support comparison. Mismatched denominators or unlabeled cost bases can mislead.
A brand analyst preregisters a test. What is recorded in advance?
- A guarantee that the hypothesis will be supported
- A rule that exploratory findings must never be mentioned
- The result the experiment is required to produce
- Hypotheses, outcomes and the analysis plan (correct)
Answer: Hypotheses, outcomes and the analysis plan. Preregistration makes intended analyses explicit. Exploratory findings can still be reported as exploratory.
A brand’s best-performing segment is identified after viewing all results. How should it be treated?
- As proof of a hypothesis never specified in advance
- As a reason to omit all other segment results
- As a validated universal winning segment
- As a finding needing careful validation (correct)
Answer: As a finding needing careful validation. Post-hoc discoveries can be useful, but they need transparent labelling and validation rather than automatic certainty.
A brand wants to evaluate a metric. Which definition is most useful?
- The largest available figure
- Named numerator, denominator, scope and time period (correct)
- A number reported without its denominator
- A familiar acronym without scope
Answer: Named numerator, denominator, scope and time period. A complete definition makes the number interpretable and repeatable. Ambiguous scope can make comparisons misleading.
Want to apply this? Analytics, CRM & attribution · All ten levels