I'm a research assistant in a behavioral psychology lab, and I've been tasked with designing a new experiment to test a specific cognitive bias, but I'm hitting a wall trying to create a clean, controlled experimental design that isolates the variable of interest. My main concern is confounding variables; for instance, how do I ensure that participant fatigue or the order of task presentation doesn't muddy the results of my primary manipulation? For experienced researchers, what is your step-by-step process for brainstorming and vetting a design before you ever run a pilot? How do you effectively use power analysis to determine your sample size, and what are your strategies for building in necessary controls without making the procedure so artificial that it lacks ecological validity?
Nice topic. The key is to start with a concrete logic model: your independent variable(s), dependent measure(s), and the boundary conditions that could invalidate the result. Then design 2–3 candidate paradigms and pick one to pilot.
Brainstorming/vetting in steps: 1) state the hypothesis and what a clean null would look like; 2) list all potential confounds (fatigue, practice effects, order, stimuli familiarity); 3) sketch at least two designs (within-subject with counterbalancing, or between-subject with random assignment); 4) map each design to an analysis plan; 5) run a quick tabletop data-flow check to see if you can actually detect the expected effect.
Power analysis is where a lot of projects trip up. Use prior estimates or a pilot to set an expected effect size. For repeated-measures designs, account for correlation among measures and possible violations of sphericity; tools like G*Power, R (pwr, simr), or jam-simulations work well. Plan for dropout and multiple comparisons, and pre-register your plan when possible. As a rough rule, a 2x2 within-subject design with a medium effect (f ~ 0.25) at alpha 0.05 and power 0.80 often lands in the mid-20s to mid-30s for participants, but simulate your exact setup to be sure.
Controls: randomize task order, consider a Latin square or full counterbalancing to reduce order and fatigue effects; include rest breaks; add a neutral task or baseline; treat order and fatigue as nuisance variables in your analysis; add manipulation checks to confirm the intended manipulation actually occurred.
Ecological validity: design your task to resemble real-world decision cues and outcomes while keeping the lab environment controlled. Combine a core lab task with occasional real-world analogs (brief surveys, simulated daily decisions) to ensure findings generalize beyond the lab; pilot test with a small sample and adjust based on feedback so it doesn’t feel artificial.
Starter template you can adapt: 1) Hypothesis and variables (IVs, DVs, control vars); 2) Design choice and rationale (within- vs between-subject, randomization, counterbalancing); 3) Power analysis method and target N; 4) Analysis plan (statistical tests, planned contrasts, handling of violations); 5) Validation steps (manipulation checks, pilot plan, preregistration and data-exclusion rules) — share with your PI as a one-page plan.