From One Environment to Many: The Problem of Reproducibility of Experimental Results

Abstract

When the same experiment is carried out in a different environment, the error term not only includes the random error within a given experiment, but it also includes the additional sources of variability that are introduced by conducting the same experiment in different environments. These differences include both natural factors such as location, time or weather and other factors such as personnel or equipment necessary to carry out this experiment. By considering the effect of changing experimental environments on the reproducibility of experiments, we try to figure out in what situations the initial experimental results will likely carry over to other environments. We examine how p-value, effect size, sample size, and the ratio of the standard deviation of environment by treatment interaction and the standard deviation of experimental error interact with one another, and as a whole, affect the experiment's reproducibility. We suggest that not only p-values but also the effect sizes and the environmental effect ratio---the ratio of the standard deviation of environment by treatment interaction and the standard deviation of experimental error---should be considered when researchers are making statistical inferences. Large effect sizes and/or small ratios of the environmental effect ratio favor high probability of reproducibility. If the environmental effect ratio is too large, the reproducibility probability may be reduced to just a coin toss, and if effect sizes are small, researchers should be very cautious about making inferences about reproducibility even if the observed p-value is small and sample size is large.