If you use the same multi-item instrument in two different samples you should never assume that reliability is the same in both samples and the whole idea that multi-item measures such as questionnaires and many rating scales have fixed internal reliability is rubbish. And the sentences like “Faust’s measure of grandiose impulses has proven reliability, Cronbach alpha .93 (Machiavelli and Evans, 1992).” really ought to be banned from all respectable journals: they are almost nonsense and part of the general pretence that such measures are much more certain and similar to measures of the natural sciences than they actually are.
As with any other sample measure that (fictional!) .93 is from a sample. If the sample was large and sampled in some sensible enough way, it may be a very precise reflection of a population value but it should have a confidence interval like any other sample statistic. Sentences like that are typical in papers in which the measure is being reused in a new sample. If the new sample and that reference one are from the same population, or not radically dissimilar in their composition, then the internal reliability in this new sample may be only trivially different from that in the cited paper, though that may still be statistically significant if you have huge samples.
However, reliability differences can be quite large if you are using the measure in very different samples. A good example is using a measure in different cultures s we showed rather nicely (I think) back in the last century (Evans, Dolan & Toriola (1997). Detection of intra- and cross-cultural non-equivalence by simple methods in cross-cultural research: evidence from a study of eating attitudes in Nigeria and Britain. Eating and Weight Disorders, 2, 67-78). Another example is using a measure of something gender sensitive like body image in men and women.
In these situations, we should always compare the sample internal reliability values. A good diagnostic check that will detect problems is to test the Cronbach alpha values in each sample. Pretty much all statistics packages will compute alpha, and SPSS has actually given it a 95% confidence interval for ages now. (It’s the CI of the “two-way mixed, consistency” ICC in RELIABILITY.) However, as far as I know, still none of the major packages offers a test of the difference between two alpha values despite Feldt having worked out the parametric statistics of the test in 1969 (see readable summary in Feldt, Woodruff & Salih (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11(1):93-103).
To remedy this, ages ago, I created an online form to carry out Feldt’s test. I haven’t replaced that with a shiny app yet. However, I think getting confidence intervals around observed alpha values is generally more useful and I do have a shiny app for that.
Both forms use R to do the calculations and use Feldt’s parametric methods, i.e. assume Gaussian distributions. For the last few years I have used R to get non-parametric bootstrap confidence intervals (and the bootstrap CI for the difference between alphas in two independent samples if I have the data for both) rather than these methods. However, from quite a lot of those computations to date, I’ve always found, the differences between the bootstrap and the parametric intervals have been small and the great advantage of the parametric approach is that you only need the observed, or reported, alpha value, the n and the number of items.
These are links to some things from my SAS days, now essentially only of historical interest I think, but using Feldt’s methods.
- feldt1.sas for CI around a single observed Cronbach alpha.
- feldt1.ssc similar program for S+ (this was the precursor of my R code)
- feldt1-table.ssc silly little extension of the above to deal with tables of alphas easily
- feldt2.sas which compares two values for independent samples and the same number of items (same computation as online form above)
- feldt2.ssc similar program for S+/R (essentially same code as in online form above!)
- feldt3b.sas which will accept a listing of pairs of values to be compared
Information from old psyctc.org site created 09.x.14, relocated and rephrased here 3.i.19, updated 7.xi.24. All content is made available under a Creative Commons License. Please feel free to reuse anything here but respect the licence, i.e. give attribution back to here.