Design effect

The design effect is formally defined as the ratio of the variance of an estimator under the actual sampling design to the variance of the same estimator under a simple random sample of the same size.¹ For a mean or proportion, it is often expressed as deff = Var_design(θ̂) / Var_SRS(θ̂). A value of 1 means the design is as efficient as simple random sampling; values above 1 indicate that the design inflates variance, often due to clustering or unequal weighting, while values below 1 indicate variance reduction, typically from stratification.² The effective sample size is then n_eff = n / deff, representing the equivalent number of independent observations.

The design effect can be decomposed into components attributable to clustering, stratification, and weighting. For a clustered design, the design effect is approximately 1 + (m − 1)ρ, where m is the average cluster size and ρ is the intraclass correlation coefficient.³ Stratification generally reduces the design effect below 1, while weighting, especially with highly variable weights, increases it. The Kish approximation for weighting is deff ≈ 1 + CV², where CV is the coefficient of variation of the weights. In practice, design effects are estimated from survey data using replication methods such as jackknife or bootstrap.

Design effects are used to plan sample sizes in complex surveys, ensuring that the sample is large enough to achieve desired precision after accounting for design inefficiencies. For example, in cluster surveys, the required sample size is multiplied by the anticipated design effect. They are also used to compute effective sample sizes for power analysis and to adjust standard errors in statistical software. In public health, the design effect is critical in surveys like the Demographic and Health Surveys (DHS) and the Multiple Indicator Cluster Surveys (MICS), where clustering is common.⁴ Reporting design effects allows readers to assess the precision of estimates from complex samples.

The design effect concept was formalized by Leslie Kish in 1965, but its roots trace to earlier work on cluster sampling.¹ A less-known variant is the 'design effect for totals' which differs from that for means when cluster sizes vary. Also, the design effect can be less than 1 for stratified samples, but this is often overlooked in practice. In longitudinal surveys, the design effect can change over waves due to attrition and reweighting. Another edge case: when weights are correlated with the outcome, the design effect can be misleadingly low, a phenomenon discussed in the literature on weight trimming. The design effect is also used in meta-analysis to adjust for clustering in multi-site trials.

Simple random sample

A sample in which every subset of the population has an equal chance of selection.

Cluster sampling

A sampling design where the population is divided into groups (clusters) and a random sample of clusters is selected.

Stratification

Dividing the population into homogeneous subgroups and sampling from each.

Intraclass correlation

A measure of the degree of similarity within clusters.