HBR's 10 Must Reads 2016

Repeating an expensive test is usually impractical, but companies can verify results in other ways. Petco sometimes deploys a staged

from affecting an experiment, and it evenly spreads any remaining (and possibly unknown) potential causes of the outcome between the test and control groups

Randomization plays an important role: It helps prevent systemic bias, introduced consciously or unconsciously,

companies typically have to make trade-offs between reliability, cost, time, and other practical considerations.

In the previous section we described the basics for conducting an experiment.

(Full disclosure: Jim Manzi’s firm, Applied Predictive Technologies, sells one, Test & Learn.)

The required sample size depends in large part on the magnitude of the expected effect.

deal with environments of high causal density, companies need to consider whether it’s feasible to use a sample large enough to average out the effects of all variables except those being studied

Although the process of experimentation seems straightforward, it is surprisingly hard in practice, owing to myriad organizational and technical challenges. That is the overarching conclusion of our 40-plus years of collective experience conducting and studying business experiments at dozens of companies, including Bank of America, BMW, Hilton, Kraft, Petco, Staples, Subway, and Walmart.