EDF 5481 READINGS AND ASSIGNMENTS

SOME ISSUES IN QUESTION CONSTRUCTION

OVERVIEW

CLICK HERE TO RETURN TO YOUR SPOT IN GUIDE 5.

We distinguish between open questions, in which the respondent uses their own words to answer, and closed questions which provide pre-written response categories.

CLOSED QUESTIONS:

• standardize responses. In carefully-written closed questions, the question and responses mean the same thing to nearly all respondents.
• are faster to administer.
• are easier to code and score.

• may suggest answers.
• have multiple meanings.
• may omit important responses.

• take longer to administer.
• may produce responses that are idiosyncratic and difficult to code into categories.

• allow a more exhaustive list of response possibilities.
• alert us to the multiple dimensions in a topic.
• convey a richness of feeling often missing from closed questions.

Closed questions should be unidimensional:  they ask about one and only one topic at a time. Questions which use more that one dimension are called “double-barreled”.

Rule 1: Avoid double-barreled questions. We cannot disentangle which question embedded in a question the respondent actually answered. Double-barreled questions are usually quickly recognized by their use of “and” or “or”. But any question that simultaneously asks about at least two topics is double-barreled.

TWO BAD DOUBLE-BARRELED EXAMPLES:

   Do you agree that we should lower property taxes AND provide more county services?

  Do you approve or disapprove of abortion in cases of incest OR threats to the mother’s health?

I have seen questions so convoluted that they were actually quadruple-barreled items, asking FOUR questions in one!

Rule 2: PROVIDE ALL RESPONSES TO A CLOSED QUESTION.

• Responses should be mutually exclusive. A respondent’s answer should fit in only ONE category.
• Responses should also be exhaustive: all possible responses are provided, even if you need to add an “OTHER. PLEASE SPECIFY:” alternative (or a don’t know or a refusal category).
• When you actually write your question, using “or something else” usually serves as the “OTHER” prompt.

EXAMPLE: For marital status, people are married, widowed, divorced, separated, living together, single (NEVER MARRIED) or something else (you’d be surprised), not just married or single.

Rule 4: Avoid condensing numerical responses into grouped categories.

• Find out education in years, not grouped degree levels. (Want degree level? ASK ABOUT degree level!)
• Ask about religious service attendance in times per month.

The exception: income categories. We use grouped categories to provide greater confidentiality in the answers and because people generally only know their incomes around April 15.

Rule 5: Use a mix of question formats to avoid format response sets or response effects.

For example, one of the most popular formats is the Likert item: people are asked whether they strongly agree, agree, are undecided, disagree, or strongly disagree with an attitude statement. Many survey researchers love Likert items because you can administer them quickly (a well-trained interviewer can do five per minute) and they are easy to code. BUT:

Don’t just use Likert agree-disagree questions. This encourages acquiescence response set. And, acquiescence response set is negatively related to education. College educated respondents are "nay-sayers."
 

RESPONSE SETS

In acquiescence response set, people tend to generally agree or disagree with nearly every sweeping attitude statement put before them in a questionnaire. Well-educated persons have been taught "never say 'always' or 'never,' " so "nay-saying," or disagreement, rises with education, typically regardless of content. We are less clear about which factors nudge people to gravitate toward extreme responses ("strongly agree" or "strongly disagree") or toward the middle, but we know that it happens too.

The tendency to simply respond "I don't know" is also linked to education. In a recent article, I found that people who had not graduated high school were three times as likely to give "I don't know" responses as people who had graduate school experience. Women were nearly twice as likely as men to say "I don't know." In such cases, you need to reflect on whether a "don't know" response truly represents lack of knowledge or some other factor (e.g., low self-efficacy).

BEWARE: "Standardized tests" which contain long sets of items, all with identical response formats, are especially prone to response sets. This is especially true if all items take a Likert-type format. In addition, "personality inventories" which contain long sets of identically phrased items (e.g., "A lot like me," "Somewhat like me," "A little like me," or "Not at all like me") are also subject to response format effects. Unfortunately, a definite, but unknown, specific amount of "reliability coefficients," such as coefficient alpha, is undoubtedly due to the fact that correlations among the items occur because of response set. Interesting...but probably not what you hoped for.

WHAT YOU CAN DO: Vary the item format that you use. Keep Likert items to a minimum. Try to include reversed items, or the same concept using different formats, in your questionnaire. Pilot test your questionnaire ALOUD even if you plan it to be self-administered. See if you detect respondents consistently falling into response patterns ("yep..." "yep..." "yep..") and "clean up" the problem areas.

Using experimental "split-ballot" techniques, survey researchers also assess whether different forms of a question yield comparable responses and appear to be comparable in meaning. Subsamples from the total survey are selected at random to receive the "same item" with different question wording. We check not only to see whether the univariate frequency distributions are the same, but also whether question wording changes how one concept relates to other variables, such as educational level.
 

Rule 6: IN GENERAL: don’t use hypothetical situations. Don't ask respondents to guess how other people “would feel” or even how they would feel under hypothetical conditions. The answers are generally unreliable because people have not thought about their responses. Most people would have a hard time telling you exactly how they would feel “if” they found out their spouse was unfaithful--except “bad” (and we don’t need a survey to figure that one out).

Rule 7: Try to keep the number of response alternatives that are read to respondents to a maximum of seven. You cannot use show cards in a telephone survey. The respondent must be able to memorize the alternatives, then select one. The fewer the response categories, the easier this is.

Rule 8: Use specific time frames when you ask about behaviors, particularly regular or habitual behaviors. Don’t leave the time frame vague or undefined if at all possible.

     “During the last month, how many times did you attend religious services?”
     “During the last week, did you smoke any cigarettes at all?”

Rule 9: Similarly, use specific place frames. Do you want to know where someone was born? (In Detroit, in a hospital, in the United States, under the kitchen table.) If you want to know someone's country of birth, ask "In which country were you born?"

Rule 10: Make sure to make the question stem consistent with the provided responses.

EXAMPLE: If the stem reads “how often”, make sure the responses are in a time frame (times per month) or take a relative form such as “All of the time”“Most of the time”“Half the time”“Seldom” or “Never”.

Reserve “yes/no” questions (“...did you smoke any cigarettes at all?”) for specific actions or dimensions of an issue.
 

Rule 11: If you have a very complicated question stem, break the question into AT LEAST two questions. The respondent will have an easier time and the questionnaire will actually go faster.

     BAD QUESTION: “What do you think should be done about the environment? Tell me all the actions that you approve: A. Recycling B. Start carpools C. Mandatory thermostat controls (etc.)”

    BETTER SET OF QUESTIONS:

• In general, do you approve or disaprove of recycling newpapers?
• In general, do you approve or disapprove of starting work carpools?
• In general, do you approve or disapprove of mandatory thermostat controls
• etc.

REMEMBER! Approve-disapprove response categories rank responses from highest to lowest (approve is more in favor than disapprove) and are ORDINAL-level variables.

Rule 12: Avoid jargon or technical terms. Your respondent probably won't know what "trait anxiety" is, even if she or he has a lot of it. "ET" to most people means a movie about a quaint alien, not Educational Technology.

Rule 13: Avoid "red flag" words, that is, words with emotional connotations or that coincide with strongly-held values.

Everyone wants to be "fair"! NEVER use that word in a question unless you are talking about a civic event or festival such as the North Florida Fair!

"Murder" is another red flag word (as in "Do you approve of the murder of unborn babies?")
 

You will not receive unbiased answers unless you ask unbiased questions.

Rule 14: If the questionnaire will be administered by an interviewer, be sure to read the entire questionnaire aloud in a pilot test. Many words sound alike that have different meanings. Consider:
 

Rule 15: Similarly, beware of words that have multiple meanings, for example: kind; fair; item. Put the Thesaurus in your computer's word processor to use!

Rule 16: In fact, it is an excellent idea to pilot test your questionnaire aloud no matter what. This is one of the best ways to catch any problems with question wording.
 
 

Rule 17: In an open-ended question, you want respondents to speak their minds.
So ask open-ended questions in a way that encourages people to give a complete and full answer.

     Use phrases such as “what are”? “how do you feel about”?
     Never ask a “yes/no” open-ended question!

     Good: “what do you like best about high school cafeteria lunches?”
     Bad: “do you like high school cafeteria lunches?”

     Good: “what do you think is the biggest issue with homeless people in Tallahassee?”
     Bad: “is homelessness a problem in Tallahassee?” (what is “problematic?”)

Rule 18: With a complex question that could have multi-dimensional answers (“what do you see as the top priorities for the Florida legislature this year?”), use an open-ended format. There will be so many possible responses you will not be able to specify all the response categories in advance.
 

September 25, 2001.
This page was built with Netscape Composer.
It is best displayed in Netscape Navigator,
600 X 800 display resolution.
Susan Carol Losh

Many jobs are available for people skilled in survey methodology.
 

EDF 5481 READINGS AND ASSIGNMENTS

OVERVIEW