Perils of the random experiment.

Most medical research is executed on samples selected from large populations. Nevertheless, health care researchers often blur the difference between interpreting sample-based research and evaluating research that included the entire population of interest. This is an implication-critical distinction; in population research, every result applies to the population (because the entire population was included in the analysis), although only a few results from sample-based research can be extended to the population at large. Treating every result from sample-based research as if that result applies to the population is misleading. Using nonmathematic terminology, this article develops the reason for the differences in the implications of these two research perspectives. In sample-based research, the best indicators of which results should be extended from the sample to the population are the presence of (1) a prospective plan for that experiment; and (2) the execution of the experiment according to that plan (concordant execution). The absence of these two features produces execution and analysis decisions based on the incoming data stream-the hallmark of the random experiment. In this latter paradigm, allowing the data to influence the execution and analysis decisions renders the usual estimates of effect size, standard errors, confidence intervals, and P values untrustworthy. Readers of clinical trial results must be vigilant for nonprotocol-driven research and understand that the results from these programs are at best exploratory and cannot be used to answer scientific questions.