Discussion questions#
When you think an effect is ‘real’ in data, do you care more that its significant, or that it’s stable and repeatable? Is an effect still meaningful or useful if it’s statistically significant but unstable? If you had to explain your results to someone non-technical, would you focus more on whether an effect exists or how reliable it is?
How do you think about what can and can’t be shuffled when building a permutation test? Where do you see parallels between statistical decisions (like what to permute) and experimental design decisions?