Bootstrap & Permutation Tests
Resampling-based hypothesis testing methods that make minimal distributional assumptions.
Test Overview & Resampling Rationale
Bootstrap Test:
Uses resampling of the observed data to create a bootstrap distribution under the null hypothesis.
Permutation Test:
Randomly reassigns group labels to test whether group differences could occur by chance.
Advantages:
- • Distribution-free
- • Exact p-values possible
- • Robust to outliers
- • Flexible test statistics
When to Use:
- • Small sample sizes
- • Non-normal distributions
- • Complex test statistics
- • Violated assumptions
Resampling Statistic & p-value Formula
Bootstrap Procedure:
- Calculate observed test statistic
- Resample data under null hypothesis
- Calculate test statistic for each bootstrap sample
- Compare observed statistic to bootstrap distribution
Permutation Procedure:
- Randomly permute group labels
- Calculate test statistic for each permutation
- Determine proportion of permutations ≥ observed
Resampling p-value (one-tailed)
= number of bootstrap/permutation resamples
T*b
= test statistic computed on resample b
Tobs
= test statistic computed on the original data
Add 1 to numerator and denominator (Davison & Hinkley correction) to avoid a p-value of exactly zero.
Key Assumptions
- • Observations are exchangeable under the null hypothesis (permutation test)
- • Sample is representative of the population (bootstrap test)
- • Resamples are drawn independently with replacement (bootstrap) or without replacement of labels (permutation)
- • Sufficient resamples are drawn for stable tail-probability estimates
Reading the Bootstrap Resampling Distribution
Example Output & Result Interpretation
Two production lines are compared on cycle time using a permutation test with 10,000 resamples. The observed difference in means is 3.2 seconds; only 210 of the 10,000 permuted differences equal or exceed 3.2 seconds.
Worked example p-value
210
= permutations with a difference ≥ the observed 3.2 s
10,000
= total permutations drawn
Common Applications:
- • Two-sample mean comparisons
- • Correlation significance testing
- • Regression coefficient testing
- • Complex multivariate comparisons
Number of Resamples:
- • Minimum: 1,000 for crude estimates
- • Standard: 5,000-10,000 for most applications
- • High precision: 50,000+ for publication
Significant Result (p ≤ α)
Observed statistic is more extreme than the resampling distribution would predict by chance — reject the null hypothesis.
Non-Significant Result (p > α)
Observed statistic is consistent with chance variation under the resampling distribution — fail to reject the null hypothesis.
sixsigmacenter.com Implementation Guide
Example Data Layout for Bootstrap/Permutation Test
Two groups for bootstrap/permutation testing. Arrange data in separate columns for each group. Sample Size Recommendation: No minimum required, but at least 5-10 observations per group recommended for meaningful resampling distributions.
This is how your spreadsheet or CSV file should be structured before you upload it. Match the column names, keep one row per observation, and leave the analysis to the platform.