This relates to the “multiple tests problem/issue”: if you do multiple statistical tests in the null hypothesis statistical testing (NHST) paradigm you typically set each one to have a risk of reporting a finding on that one test as statistically significant of one in twenty (1/20 = .05). That’s the probability of seeing a result as strong or stronger than you did on that one test if in the population from which you took your sample actually there is no pertinent effect. When you do lots of tests each one may have a risk of a “false positive” of one in twenty but the risk that you have at least one false positive goes up rapidly with the number of tests. This is the “multiple tests issue/problem”. There are ways, such as the Bonferroni procedure, of keeping the risk of any (i.e. one or more) falsely positive findings at .05 but they mean your probability of finding a statistically significant effect drops assuming the same real population effect and sample size.
Details #
The FDR is just the proportion of the tests that will be come out as statistically significant which were false. So say you do 50 tests and 10 come out as statistically significant on some FDR managing test approach then you can set the proportion you will tolerate being false. If you set the FDR your testing should use to .1, i.e. one in ten, then in principle, i.e. in the long run, if you keep using the approach again and again and again, .1, 10%, one in ten of those statistically significant findings will be false. FDR controlling methods don’t reduce the statistical power of your tests as much as the Bonferroni type methods.
Of course, there is no magic that can tell us which of our statistically significant tests are the false ones whatever procedures we use to minimise (Bonferroni type) or control (FDR type) their rate.
These methods really came into prominence and wide usage as the computer power to do them arrived and as things like testing for associations with, say, many genetic loci or blood markers or brain scan findings also arrived. I have yet to see them used in therapy research but I do see papers where the multiple tests issue, and its risk of creating many false positives are not discussed at all and sometimes I’m in that situation: we have many associations we can explore in a set of data and Bonferroni style approaches will drop our power to detect effects to near zero given our small sample sizes. I don’t think there are any perfect answers but the issue does need to be acknowledged much more than it is in our field and perhaps FDR methods might be helpful for us
Try also #
Multiple tests issue/problem
Bonferroni correction
Null hypothesis testing
Null hypothesis significance testing (NHST) paradigm
Population
Sampling and sample frame
Chapters #
Not in the OMbook.
Online resources #
None yet.
Dates #
First created 28.x.24.
