Reliability

Reliability is the extent to which any measuring procedure gives similar results with repeated use with the same respondents. In actual practice, we do not expect to get exactly the same results with repeated use, but we do expect to see a high degree of consistency from one use of a measuring instrument to the next. Persons scoring high one time would be expected to score high again, give or take a little. Those scoring low or in between would be expected to score about the same way on repeated measurement. Generally, a coefficient of correlation is used to establish the reliability of a measuring instrument (see Chapter 19). Several methods can be used to test for reliability.

Test-retest method

An obvious way to test for consistency is to get responses for an indicator on one occasion and then at a second time. The same sample and the same measure are used each time. A test of the relationship between the two sets of scores is used to establish the degree of consistency or reliability of the measuring instrument. Returning to our earlier example about attitudes toward male/female equality, we could ask a sample of couples to answer a set of questions about equality in marriage and then return several weeks later and have them respond to the same questions again.   

While easy to use, the test-retest method has several limitations. Many respondents may think answering the same questions a second time is foolish. Some may even refuse. For those who do cooperate a second time, some may consciously try to respond the way they did the first time. This could create greater consistency in responses than is actually true. Another problem is that some event may occur between the two times of measurement that could influence how they respond the second time. In our illustration of attitudes toward male/female equity in marriage, for example, some crisis in a family could change how a husband or wife would answer. This danger increases as the time interval between measurements increases. To get around this limitation, another method is sometimes used.

Equivalent forms method

In this method, the investigator develops two equivalent forms of the measuring instrument. The items are combined in one set and respondents respond to all the items at the same time. A statistical test is generally used to estimate the degree of consistency in answering items making up each form of the measure.  

The equivalent forms method, also known as the split-half method, has the advantage of ruling out the effects of the first measurement, such as memory or the influence of external events, both of which affect the validity of the test-retest method. This method, however, has three disadvantages. First, the researcher has to develop twice as many carefully constructed and tested items. Most of us have difficulty developing one set of useable items, let alone two. Second, there is always some doubt about how equivalent any two forms are. To protect against their obvious interest in claiming equivalence, investigators sometimes ask a panel of judges to rate the equivalence of items. Using judges certainly strengthens claims for this approach, but still relies on judgment instead of any external empirical evidence. Third, research has shown that different reliability results are obtained depending on how the two halves are created. Another method, described next, overcomes these limitations.

Chronbach's alpha

Many researchers use a method introduced by Chronbach (1951) in calculating the reliability of a set of items. This method requires understanding how to use and calculate correlation coefficients (see Chapter 19). Chronbach's alpha is based on calculating the correlation coefficient for relationships between each item and every other item in the set making up a measuring instrument. Even with a small number of items, this method involves a number of calculations. For instance, for a scale with 10 items, 45 inter-item correlations would have to be calculated. If you don't have access to a statistical package on a personal computer, calculations of the inter-item relations will require a large amount of work. With a statistical package, the work is greatly reduced.