Experimental Design for GEORepeated Runs, Paraphrases, and Factorial Controls

Why generative variability demands repeated observations, intent-preserving paraphrases, and structured factorial designs.

Simple before-and-after designs confound treatment with time

The difference between one pre-treatment week and one post-treatment week contains the intervention, platform drift, competition, and random noise. More observations improve precision but do not repair a design that lacks a credible comparison.

Repeated runs estimate a distribution

One prompt can yield different sources, brands, citations, and ordering across executions. Repeated runs reveal average outcomes, variance, flip rates, and uncertainty. The required number depends on baseline variance, target effect size, clustering, and the cost of false conclusions.

Prompt paraphrases are experimental controls

Several formulations of the same information need test whether an effect survives wording changes. “Best CRM,” “recommend a CRM,” and “which CRM should I choose?” should be treated as related variants—not pooled as independent evidence without preserving their intent group.

Factorial designs expose interactions

A factorial design can cross treatment with engine, prompt family, paraphrase, and time. It estimates not only an average effect but whether the intervention behaves differently across platforms or information needs.