WLA 2008: Constructing Questionnaires and Questions: It's Harder Than it Seems

This presentation by Thomas D. Walker (UW Milwaukee) covered the fundamentals of questionnaire construction.

Surveys are instruments to gather data for empirical analysis.

Some questionnaires can accidentally turn into "fishing expeditions", and ask too many questions.

There are three crucial tasks:

• sampling the population
• determining the medium - f2f 1 to 1, in groups, written form
• designing the questionnaire

Advantages to written Questionnaires:

• more cost effective than interviews, especially with large sample sizes or large regions
• most cost effective medium for large numbers of questions
• can be easier to analyze, depending on the construction of the questions; some aspects can be automated (#2 pencil)
• are a familiar medium
• can reduce bias associated with oral questionnaires
• can be less instusive than oral surveys (more anonymous)

Disadvantages to written questionnaires:

• lower response rates (equals lower confidence levels)
• questions are usually very focused -- no elaborations are possible ("well yes, but ..." qualified answers aren't possible)
• no visual cues (no body language observable by interviewer)
• hard to know who's fillig it out
• may not be suited to certain populations (jargon, reading level issues, language, etc.)

Developing a questionnaire:

• Define:
• what kind of info is required?
• from whom do you need data?
• Break down complex problems into very simple ones

Questionnaires:

• should be introduced to let the respondent know what the purpose of the survey is, who will analyze it, and whether the results will be made public
• should conclude by expressing appreciation
• should be designed at a relatively early reading level

Good questions are...
clear
concise

Non-threatening questions...

• seek truth on sensitive issues by using a clinical, anonymous distance
• gain the confidence of the subject
• do not lead in one direction or another

Seek one kind of info, not two or more:
"Are you satisfied with the hours and facilities of the library?"

If choices are provided, ensure they include all possibilities.
"Which of the following services do you use? -ILL - Reference -YA"

Offer mutually-exclusive choices; don't provide choices that overlap

Logical sequence

• group questions logically
• establish a logical flow within a group
• possible characteristics: general to specific; positives versus negatives; time sequence
  

Rewording to validate

• Some large-scale studies may benefit from the validation of data by means of question repetition
• Most of the time, surveys we do aren't this large

Make no assumptions
"How satisfied are you with DPL's provision of access to large-scale bibliographic databases?"

Does not suggest an answer; doesn't lead the person to an answer

Avoid jargon and acronyms

Should not be tied to other questions; avoid especially in written questionnaires

Adhere to the Rule of 5

• Likert scale 1 - 5
• Rankings

Direct questions:

• True / false
• Multiple choice
• Likert scale
• Ranking

Advantages to direct/closed questions:

• easy to answer
• easy to code
• responses are uniform
• success of closed quesitons depends on the quality of the questions

Scaling

• Likert scale
• spectrum between two poles

Open-ended questions

• unstructured
• sentence completion
• word association

Advantages of open-ended questions:

• more flexible
• richer data (hard to analyze)
• may lead to other variables (other questions you want to ask in a future questionnaire)

Disadvantages of open-ended questions:

• hard to code/analyze

Confidentiality:

• statements assuring confidentiality are desirable and may be required
• inform respondents that thier responses are voluntary and their anonymity is assured
• if children are involved in any way, extra precautions must be taken

Question order:

• don't start with sensitive questions
• lead logically and unthreateningly to sensitive questions
  
• request demographic data at the end
  

Return rates from online surveys of distance education students are dismally low.