The Response Rate Conundrum in Survey Research

Response rates are suggested to be a critical indicator of survey and response quality. Thus research papers are expected to report response rates. However, this step is not as easy as it seems.

Response rate is defined as the percentage of the eligible sample that responds to the survey. As this definition indicates, how large is the eligible sample is an important criteria in this calculation.

Some texts and research papers suggest that non-contactable respondents be considered a part of the eligible sample. Thus,

Response Rate = Responses / Eligible Sample

where Eligible Sample = Responses + Refusals + Non-contacts

However, this is not true in several contexts. For example, making contact may be the only means by which one can establish the existence of a potential respondent. Or making contact may be the only way to determine eligibility. In such situations many papers define the eligible sample as responses plus refusals. This can plausibly lead to overstating the response rate.

Thus the response rate conundrum can be expressed as a range of response rates that lie with the following range:

Response Rate (Lower Bound) = Responses / (Responses + Refusals + Non-contacts)

Response Rate (Upper Bound) = Responses / (Responses + Refusals)

The true value of the response rate would lie near:

Response Rate (Likely) = Responses / (Responses + Refusals + EE(Non-contacts))

Here EE is Estimated Eligibility of non-contacts, i.e. the estimated proportion of non-contacts that would have been eligible. One way of calculating EE is by dividing the sum of responses and refusals (which is the determined eligible sample) by the number of contacted potential respondents.

An illustrative example is given below:

If:

Non-verified Sample Pool: 100

Contacted Respondents: 50

Verified Sample: 25

Responses: 15

Refusals: 10

Thus:

Response Rate (Lower Bound) = 15 / (15 + 10 + 50) = 5%

Response Rate (Upper Bound) = 10 / 25 = 40%

Response Rate (Likely) = 10 / (15 + 10 +(25/50)*50) = 20%

One thought on “The Response Rate Conundrum in Survey Research

  1. Pingback: Response Rates in Organizational Surveys: How Much is Enough? « A little bit of this, a little bit of that

Leave a comment