Chapter 2: Research Design
Threats to internal validity
There are many different ways that the internal validity of a study can be threatened or jeopardised. A list and brief comment of some of the more important ones are given below.
Selection bias. Occurs when more of one type of person gets into one group for a study. For example, the people who return your questionnaire may be different, in some important way, to the people who did not return your questionnaire. The students who volunteer for your project might be different to the ones who do not volunteer (for example, more altruistic, more achievement oriented, more intelligent). Do these variables have an effect on the thing you are trying to measure? We usually do not know.
Drop-out. More of one type of person may drop out of one of the groups. For example, those less committed, less achievement-oriented, less intelligent.
History. Events that happen to participants during the research which affect results but are not linked to the IV. In an extended study comparing relaxation to no relaxation on headache occurrence, those in the no relaxation condition sought out other means of reducing their headache occurrence (e.g. took more pills).
Reliability of measures and procedures. Unreliable operationalisations of constructs, or inconsistency in giving instructions to participants, or training to assessors can invalidate the study.
Using a design of low power. In particular, a small sample size may have insufficient power to detect a real effect even if it is there. As a result, the researcher claims the manipulation had no effect when in fact it does; he just couldnŐt pick it up. As well, different statistical tests have varying sensitivity to detect differences.
Order effects. If we measure something over a series of trials, we might find that a change occurs because our participants are becoming bored, tired, disinterested, fatigued, less motivated than they were at the beginning of the series. "Counterbalancing" is a way of overcoming this problem in repeated measures designs.
Multiple tests of significance. The more significance tests (Chapter 6) you conduct on the one set of data, the more likely you are to claim that you made a significant finding when you should not have. You will be capitalising on chance fluctuations.
© Copyright 2000 University of New England, Armidale, NSW, 2351. All rights reserved
Maintained by Dr Ian Price