Holdout Groups
Holdout Groups are control segments deliberately excluded from specific marketing campaigns to measure incremental impact. The experimental approach quantifies true effectiveness by comparing exposed and unexposed audiences, delivering rigorous attribution insights.
Understanding Holdouts
Control Group Methodology
Holdout groups apply controlled experimental design to attribution:
Concepts: - Test group: Users exposed to campaigns - Control group: Users withheld from exposure - Random assignment: Statistical randomization for unbiased groups - Incremental lift: Additional conversions driven by marketing
Practical example:
Holdout experiment design:
Total Target Audience: 100,000 users
Test Group (90%): 90,000 users → Exposed to display advertising campaign
Control Group (10%): 10,000 users → No display advertising exposure
Results after 30 days:
Test Group Conversions: 1,800 (2.0% conversion rate)
Control Group Conversions: 150 (1.5% conversion rate)
Incremental Lift: 2.0% - 1.5% = 0.5%
Marketing Attribution: 0.5% / 2.0% = 25% of conversions attributable to campaign
Implementation
Group assignment:
// Holdout group management and assignment system
class HoldoutGroupManager {
constructor() {
this.activeExperiments = new Map();
this.userAssignments = new Map();
this.experimentResults = new Map();
}
createHoldoutExperiment(experimentConfig) {
const experiment = {
experiment_id: this.generateExperimentId(),
name: experimentConfig.name,
description: experimentConfig.description,
start_date: experimentConfig.start_date,
end_date: experimentConfig.end_date,
holdout_percentage: experimentConfig.holdout_percentage,
target_audience: experimentConfig.target_audience,
campaign_details: experimentConfig.campaign_details,
status: 'active',
randomization_seed: Math.random().toString(36).substr(2, 9)
};
this.activeExperiments.set(experiment.experiment_id, experiment);
// Pre-assign users to test and control groups
this.assignUsersToGroups(experiment);
return experiment;
}
assignUsersToGroups(experiment) {
const targetUsers = this.getTargetAudience(experiment.target_audience);
targetUsers.forEach(userId => {
const assignmentHash = this.generateAssignmentHash(
userId,
experiment.experiment_id,
experiment.randomization_seed
);
// Use hash-based assignment for consistent randomization
const hashValue = parseInt(assignmentHash.substr(0, 8), 16) / 0xffffffff;
const isHoldout = hashValue < (experiment.holdout_percentage / 100);
const assignment = {
user_id: userId,
experiment_id: experiment.experiment_id,
group: isHoldout ? 'control' : 'test',
assignment_timestamp: Date.now(),
hash_value: hashValue
};
this.storeUserAssignment(assignment);
});
}
shouldServeMarketing(userId, campaignId) {
const relevantExperiments = this.getExperimentsForCampaign(campaignId);
for (let experiment of relevantExperiments) {
const assignment = this.getUserAssignment(userId, experiment.experiment_id);
if (assignment && assignment.group === 'control') {
// User is in holdout group - do not serve marketing
this.logHoldoutEvent(userId, campaignId, experiment.experiment_id);
return false;
}
}
return true; // Serve marketing to test group and non-experiment users
}
trackConversion(userId, conversionData) {
// Record conversion and associate with relevant experiments
const userExperiments = this.getUserExperiments(userId);
userExperiments.forEach(experiment => {
const assignment = this.getUserAssignment(userId, experiment.experiment_id);
if (assignment) {
const conversionEvent = {
user_id: userId,
experiment_id: experiment.experiment_id,
group: assignment.group,
conversion_value: conversionData.value,
conversion_type: conversionData.type,
conversion_timestamp: Date.now(),
days_since_assignment: this.daysBetween(
assignment.assignment_timestamp,
Date.now()
)
};
this.recordExperimentConversion(conversionEvent);
}
});
}
}
Statistical analysis:
# Holdout experiment statistical analysis
import scipy.stats as stats
import numpy as np
import pandas as pd
class HoldoutAnalyzer:
def __init__(self):
self.confidence_level = 0.95
self.minimum_sample_size = 1000
self.minimum_runtime_days = 14
def analyze_experiment_results(self, experiment_id):
experiment_data = self.get_experiment_data(experiment_id)
# Group-level analysis
test_group = experiment_data[experiment_data['group'] == 'test']
control_group = experiment_data[experiment_data['group'] == 'control']
analysis_results = {
'experiment_id': experiment_id,
'analysis_timestamp': datetime.now(),
'statistical_significance': {},
'incremental_lift': {},
'group_statistics': {},
'recommendations': []
}
# Calculate basic statistics
analysis_results['group_statistics'] = {
'test_group': {
'size': len(test_group),
'conversions': test_group['converted'].sum(),
'conversion_rate': test_group['converted'].mean(),
'total_value': test_group['conversion_value'].sum(),
'average_value': test_group['conversion_value'].mean()
},
'control_group': {
'size': len(control_group),
'conversions': control_group['converted'].sum(),
'conversion_rate': control_group['converted'].mean(),
'total_value': control_group['conversion_value'].sum(),
'average_value': control_group['conversion_value'].mean()
}
}
# Statistical significance testing
analysis_results['statistical_significance'] = self.calculate_significance(
test_group, control_group
)
# Incremental lift calculation
analysis_results['incremental_lift'] = self.calculate_incremental_lift(
analysis_results['group_statistics']
)
# Generate recommendations
analysis_results['recommendations'] = self.generate_recommendations(
analysis_results
)
return analysis_results
def calculate_significance(self, test_group, control_group):
# Conversion rate significance test
test_conversions = test_group['converted'].sum()
test_size = len(test_group)
control_conversions = control_group['converted'].sum()
control_size = len(control_group)
# Two-proportion z-test
z_stat, p_value = stats.proportions_ztest(
[test_conversions, control_conversions],
[test_size, control_size]
)
# Conversion value significance test (t-test)
if test_conversions > 0 and control_conversions > 0:
test_values = test_group[test_group['converted'] == 1]['conversion_value']
control_values = control_group[control_group['converted'] == 1]['conversion_value']
t_stat, t_p_value = stats.ttest_ind(test_values, control_values)
else:
t_stat, t_p_value = None, None
return {
'conversion_rate_test': {
'z_statistic': z_stat,
'p_value': p_value,
'is_significant': p_value < (1 - self.confidence_level),
'confidence_interval': self.calculate_confidence_interval(
test_conversions, test_size, control_conversions, control_size
)
},
'conversion_value_test': {
't_statistic': t_stat,
'p_value': t_p_value,
'is_significant': t_p_value < (1 - self.confidence_level) if t_p_value else False
}
}
def calculate_incremental_lift(self, group_stats):
test_cr = group_stats['test_group']['conversion_rate']
control_cr = group_stats['control_group']['conversion_rate']
absolute_lift = test_cr - control_cr
relative_lift = (absolute_lift / control_cr) if control_cr > 0 else float('inf')
# Calculate incremental conversions
test_size = group_stats['test_group']['size']
incremental_conversions = absolute_lift * test_size
# Calculate incremental value
test_avg_value = group_stats['test_group']['average_value']
incremental_value = incremental_conversions * test_avg_value
return {
'absolute_lift': absolute_lift,
'relative_lift': relative_lift,
'incremental_conversions': incremental_conversions,
'incremental_value': incremental_value,
'attribution_percentage': absolute_lift / test_cr if test_cr > 0 else 0
}
def calculate_power_analysis(self, experiment_data):
"""Calculate statistical power and required sample sizes"""
from statsmodels.stats.power import ttest_power
current_effect_size = self.calculate_effect_size(experiment_data)
current_power = ttest_power(
current_effect_size,
nobs=len(experiment_data) / 2, # Approximate per-group size
alpha=1 - self.confidence_level
)
# Calculate required sample size for 80% power
from statsmodels.stats.power import ttest_power
required_total_size = 0
for n in range(100, 100000, 100):
power = ttest_power(current_effect_size, nobs=n/2, alpha=0.05)
if power >= 0.8:
required_total_size = n
break
return {
'current_effect_size': current_effect_size,
'current_statistical_power': current_power,
'required_sample_size_80_power': required_total_size,
'is_adequately_powered': current_power >= 0.8
}
Applications
Campaign Effectiveness
Display advertising holdout:
Display effectiveness analysis using holdouts shows:
graph TD
A[Total Audience: 1M users] --> B[Test Group: 900K users]
A --> C[Control Group: 100K users]
B --> D[Exposed to Display Ads]
C --> E[No Display Ad Exposure]
D --> F[Conversions: 18,000 (2.0%)]
E --> G[Conversions: 1,200 (1.2%)]
F --> H[Incremental Lift: 0.8%]
G --> H
H --> I[Attribution: 40% of conversions from display ads]Multi-channel results: | Channel | Test Group CR | Control Group CR | Lift | Attribution % | Statistical Significance | |---------|---------------|------------------|------|---------------|------------------------| | Display | 2.0% | 1.2% | 0.8% | 40% | p < 0.001 | | Video | 1.8% | 1.3% | 0.5% | 28% | p < 0.01 | | Social | 2.5% | 1.9% | 0.6% | 24% | p < 0.001 | | Search | 3.2% | 0.8% | 2.4% | 75% | p < 0.001 |
Brand Awareness Impact
Holdout for brand campaigns:
# Brand awareness holdout analysis
def analyze_brand_awareness_impact():
brand_campaign_results = {
'campaign_type': 'brand_awareness',
'flight_duration': '4_weeks',
'target_audience': 'prospective_customers',
'holdout_percentage': 20,
'test_group': {
'size': 800000,
'brand_search_lift': 0.35, # 35% increase in brand searches
'direct_traffic_lift': 0.22, # 22% increase in direct traffic
'conversion_rate': 0.018,
'average_order_value': 125
},
'control_group': {
'size': 200000,
'brand_search_baseline': 0.008, # Baseline brand search rate
'direct_traffic_baseline': 0.045, # Baseline direct traffic rate
'conversion_rate': 0.015,
'average_order_value': 123
}
}
# Calculate brand impact metrics
brand_impact = {
'incremental_brand_searches': (
brand_campaign_results['test_group']['size'] *
brand_campaign_results['test_group']['brand_search_lift']
),
'incremental_direct_visits': (
brand_campaign_results['test_group']['size'] *
brand_campaign_results['test_group']['direct_traffic_lift']
),
'incremental_conversions': (
brand_campaign_results['test_group']['size'] *
(brand_campaign_results['test_group']['conversion_rate'] -
brand_campaign_results['control_group']['conversion_rate'])
)
}
return brand_impact
# Long-term brand building measurement
class LongTermBrandImpact:
def __init__(self):
self.measurement_periods = [30, 60, 90, 180, 365] # days
def measure_brand_decay(self, holdout_experiment_id):
decay_analysis = {}
for period in self.measurement_periods:
period_results = self.analyze_period_results(
holdout_experiment_id,
period
)
decay_analysis[f'day_{period}'] = {
'incremental_lift': period_results['lift'],
'decay_rate': self.calculate_decay_rate(period_results),
'half_life_estimate': self.estimate_half_life(period_results)
}
return decay_analysis
Audience Segmentation
Segment-specific holdout:
# Advanced holdout analysis by audience segments
class SegmentedHoldoutAnalysis:
def __init__(self):
self.segments = {
'new_customers': 'users_without_previous_purchases',
'returning_customers': 'users_with_purchase_history',
'high_value': 'users_with_clv_above_200',
'price_sensitive': 'users_responding_to_discounts',
'brand_loyal': 'users_with_repeat_brand_interaction'
}
def analyze_segment_responses(self, experiment_id):
segment_results = {}
for segment_name, segment_definition in self.segments.items():
segment_data = self.get_segment_experiment_data(
experiment_id,
segment_definition
)
if len(segment_data) >= self.minimum_sample_size:
segment_analysis = self.perform_holdout_analysis(segment_data)
segment_results[segment_name] = segment_analysis
# Compare segment effectiveness
effectiveness_ranking = self.rank_segments_by_effectiveness(segment_results)
return {
'segment_results': segment_results,
'effectiveness_ranking': effectiveness_ranking,
'targeting_recommendations': self.generate_targeting_recommendations(
segment_results
)
}
def generate_targeting_recommendations(self, segment_results):
recommendations = []
for segment, results in segment_results.items():
if results['incremental_lift']['relative_lift'] > 0.2: # 20%+ lift
if results['statistical_significance']['is_significant']:
recommendations.append({
'segment': segment,
'action': 'increase_investment',
'reasoning': f"High incremental lift of {results['incremental_lift']['relative_lift']:.1%}",
'priority': 'high'
})
elif results['incremental_lift']['relative_lift'] < 0.05: # <5% lift
recommendations.append({
'segment': segment,
'action': 'reduce_investment',
'reasoning': f"Low incremental lift of {results['incremental_lift']['relative_lift']:.1%}",
'priority': 'medium'
})
return recommendations
Advanced Methodologies
Sequential Holdout Testing
# Sequential holdout experiment manager
class SequentialHoldoutManager:
def __init__(self):
self.experiment_stages = []
self.stage_transition_rules = {}
def design_sequential_experiment(self, base_config):
"""
Design multi-stage holdout experiment with adaptive allocation
"""
stages = [
{
'stage_name': 'pilot',
'duration_days': 14,
'sample_percentage': 5, # 5% of total audience
'holdout_percentage': 50, # 50% holdout for maximum learning
'success_criteria': {
'min_lift': 0.1, # 10% minimum lift
'significance_threshold': 0.1 # p < 0.1 for pilot
}
},
{
'stage_name': 'validation',
'duration_days': 21,
'sample_percentage': 20, # Expand to 20% of audience
'holdout_percentage': 20, # Reduce holdout to 20%
'success_criteria': {
'min_lift': 0.05, # 5% minimum lift
'significance_threshold': 0.05 # p < 0.05 for validation
}
},
{
'stage_name': 'full_deployment',
'duration_days': 30,
'sample_percentage': 100, # Full audience
'holdout_percentage': 5, # Minimal 5% holdout for monitoring
'success_criteria': {
'min_lift': 0.02, # 2% minimum lift
'significance_threshold': 0.05
}
}
]
return self.implement_sequential_stages(stages, base_config)
def evaluate_stage_progression(self, current_stage_results):
"""
Determine whether to proceed to next stage or terminate experiment
"""
success_criteria = current_stage_results['success_criteria']
actual_results = current_stage_results['actual_results']
meets_lift_criteria = (
actual_results['relative_lift'] >= success_criteria['min_lift']
)
meets_significance_criteria = (
actual_results['p_value'] <= success_criteria['significance_threshold']
)
if meets_lift_criteria and meets_significance_criteria:
return {
'decision': 'proceed_to_next_stage',
'confidence': 'high',
'reasoning': 'Both lift and significance criteria met'
}
elif meets_lift_criteria:
return {
'decision': 'extend_current_stage',
'confidence': 'medium',
'reasoning': 'Lift criteria met but needs more data for significance'
}
else:
return {
'decision': 'terminate_experiment',
'confidence': 'high',
'reasoning': 'Insufficient lift observed'
}
Cross-Platform Coordination
// Cross-platform holdout coordination system
class CrossPlatformHoldoutCoordinator {
constructor() {
this.platforms = ['web', 'mobile_app', 'email', 'social_media'];
this.coordinated_experiments = new Map();
this.platform_integrations = {
'web': new WebHoldoutManager(),
'mobile_app': new MobileHoldoutManager(),
'email': new EmailHoldoutManager(),
'social_media': new SocialMediaHoldoutManager()
};
}
createCoordinatedExperiment(experimentConfig) {
const coordinated_experiment = {
experiment_id: this.generateExperimentId(),
global_holdout_percentage: experimentConfig.holdout_percentage,
user_assignment_strategy: 'global_consistent',
platform_specific_configs: {},
cross_platform_attribution: true
};
// Ensure consistent user assignment across platforms
const global_user_assignments = this.generateGlobalAssignments(
experimentConfig.target_audience,
experimentConfig.holdout_percentage
);
// Configure each platform
this.platforms.forEach(platform => {
const platform_config = {
...experimentConfig,
platform: platform,
user_assignments: global_user_assignments,
platform_specific_rules: experimentConfig.platform_rules?.[platform]
};
coordinated_experiment.platform_specific_configs[platform] =
this.platform_integrations[platform].createExperiment(platform_config);
});
this.coordinated_experiments.set(
coordinated_experiment.experiment_id,
coordinated_experiment
);
return coordinated_experiment;
}
analyzeCoordinatedResults(experiment_id) {
const experiment = this.coordinated_experiments.get(experiment_id);
const platform_results = {};
let combined_test_group = [];
let combined_control_group = [];
// Collect results from each platform
this.platforms.forEach(platform => {
const platform_results_data = this.platform_integrations[platform]
.getExperimentResults(experiment.platform_specific_configs[platform].id);
platform_results[platform] = platform_results_data;
combined_test_group.push(...platform_results_data.test_group);
combined_control_group.push(...platform_results_data.control_group);
});
// Perform unified analysis
const unified_analysis = this.performUnifiedAnalysis(
combined_test_group,
combined_control_group
);
return {
experiment_id: experiment_id,
platform_specific_results: platform_results,
unified_results: unified_analysis,
cross_platform_insights: this.generateCrossPlatformInsights(
platform_results,
unified_analysis
)
};
}
}
Holdout groups offer the most rigorous method for measuring true incrementality and attribution. They need careful design and analysis, but deliver unbiased insights for budget allocation and optimization.
Recommended for organizations seeking precise ROI measurement, incrementality testing, and evidence-based optimization. Works best for campaigns with sufficient scale, statistical power, and stakeholders committed to rigorous experimentation.
About AI participation in writing articles
This article, like many others on our site, was created, written and proofread by a team of developers. Of course, not without the participation of AI assistants. We don't hide this and believe that modern systems are already quite good at handling simple tasks and, relatively speaking, writing an article about Viewport yourself is quite strange. It won't come out significantly better and will take a lot of time. But providing basic understanding to beginner webmasters is necessary. Of course, after the article is written by assistants - there's always proofreading, and this is where not one or two people participate, and only after that the article is published.
Ready to measure true marketing incrementality?
Sign up for a free trial. Access holdout group management, statistical analysis, and incrementality measurement for rigorous attribution insights.
Ready to take control of your web analytics? Try Statable free for 30 days — no credit card required, full feature access, GDPR-compliant by default. Start your free trial or view a live demo.