Aspects of Questionnaire Design

I have blogged before about using executive presentations and other artifacts as part of organization-wide organizational learning (see below), and in this blog I will cover some aspects of questionnaire design.

I will assume that the reader either already knows or will research the basic questionnaire-design constructs like having clear agent of action, avoiding double-barreled questions, speaking in the active voice, and the like.

Why Create a Quiz?

The objective of a quiz is to perplex and mystify the reader, or so you have been led to believe over the years of taking them yourself and being mystified at the purpose of some questions that seem to delve the depths of irrelevancy, and perplexed by the minutes of your life ebbing away as you answer them.

However, this is not at all the purpose of setting questions.

Setting questions achieves the following:

  • A second shot at highlighting to the audience what you regard as the important take-away points. You should set questions only on the things you want the reader to know are the most important bits – by posing a question you are saying to the audience “…and this is the important part
  • Finding out if there is something inside your material that is systematically misleading the audience. If significant numbers of people get the wrong answer, then you have misled them someplace and you need to fix that.
  • Finding out if there is a bias of some kind in the audience population. If only one department, or a specific age-group, or only people over six feet tall get certain answers wrong, or pick a specific incorrect answer from a list, then something is going on that you need to look into – which is probably something you told them previously.
  • Finding out if what you said made a lick of sense.
  • Discovering if the person felt confident about their answer or not

Of course this gets a tiny bit more complicated, but then that is why you are in this business – you like complicated things.


Which bring us to How.

Rules

  1. Only ask questions that test understanding on something you regard as a vital point – don’t waste your time and theirs on setting questions on irrelevant material.

  2. Never offer frivolous alternatives in a multiple-choice question, each alternative should be something the person is likely to pick due to a misunderstanding that you have already discovered.

  3. If in doubt, leave it out.

  4. Test and retest before launching.

  5. You need the SME to be involved in building a questionnaire because only they can know which questions are significant, and which answers are valid.

How to Create a Quiz

I have a book on my shelf that is written by the guru on questionnaire design, A.N. Oppenheim (Oppenheim1998) and one of the few books exclusively focused on the topic of designing questions. The preface to the 2nd edition starts off with the following:

The world is full of well-meaning people who believe that anyone who can write plain English and has a modicum of common sense can produce a good questionnaire. This book is not for them”

The basic drift is that it isn’t that simple to construct a good questionnaire, and boy, isn’t it in spades!

Ask a bad question, and you will get nonsensical answers and be left wondering what the audience thought you meant.
You will also have wasted your chance, and have wasted the respondent’s time – for which there is no excuse whatsoever.

There are plenty of texts (such as published by O’Reilly) dealing with the technical side of questionnaire tools both SaaS ( Survey Monkey, etc.) or embedded within Learning Management ( Moodle, WebCT, etc.), Trouble-Ticketing ( Remedy, OTRS, etc.), and other suites.
But that’s the easy part, albeit the part with the thickest manuals.

What I am going to cover here is the more tricky part of how to build the dialogue involved in asking questions in an eLearning context.
You cannot see the puzzled look on your respondent’s face in an eLearning situation, so you will have to plan for it when you design your questions.

Step 1 – Critical Elements

Identify the critical concepts or facts that you want the audience to understand and retain, jot these down.
If you get past 15 or 20, consider breaking your course into more than one part – a tutorial with more than a dozen critical points is starting to get really big, and unlikely to stick. Five is a good number, try to keep it that focused.
Keep it tight, keep it light, and rather build more tightly-focused courses than trying to solve the world’s problems in one fell swoop.

Step 2 – “By George She got it”

For each question, consider what supportive information you can give for a correct answer.
You are getting another shot at contextualizing and once more to drive a point home, don’t waste it.
You should present the respondent with a text of your choice and you should conform to the dialectic form of “
yes, and …”.
Affirm the correct answer and then provide the context of why that answer is right, and drive the point home a little deeper.

Step 3 – “um… no, because …”

For each incorrect answer you provide or which might occur (you will enable them to pick a wrong answer, right?), you need to furnish targeted corrective information.
Try to present wrong alternatives not to confuse, but to identify what you think are common mistakes or potential mistakes you want to address, so that once more you can drive your point home and provide a context.
The idea is to provide them with enough information (including referring or linking to other sources), so that you get the issue cleared up in their mind before they move on.

Step 4 – Concluding Summary

Many questionnaire tools will give you the option (which you will naturally take) to provide a feedback statement after they have finished answering it.

This is once again, an opportunity to provide additional context or remind them of the facts.
It allows you to place the question and answer in perspective in the broader picture, and provide the respondent with an additional link in why this is important and how to picture it.

Step 5 – Confidence

There is a big difference between getting something wrong when you are taking your best guess and being wrong and simultaneously being very confident about your answer, and it is very useful to know which is the case.
Consider constructing your questionnaire to add a rider to each question to measure how confident the respondent is – a simple five-point
Likert Scale should be fine.

Conclusion

By now you can see another reason why asking irrelevant questions is a waste of effort – for each question you need a comment for a right answer, comments for wrong answers, and a comment to put the whole question into a meaningful perspective.
A whole bunch of work that you only want to do if the question is worth the effort.

Remember, you are in an asynchronous dialogue with the respondent, and the objective is to pass on not just facts, not merely information, but knowledge – and you can only do that by also providing perspective and context.

Good Luck!

Some previous posts on Organizational Learning:

“How to get added value from corporate presentations”

“Knowledge Transfer”

“Niche Mastery – How KM can add a few hundred million dollars to corporate worth”

“The Corporate College is dead, long live the Corporate College!”

“Corporate blogging and web2.0 – training wheels first

Please contribute to my self-knowledge and take this 1-minute survey that tells me what my blog tells you about me. – Completely anonymous.

Bibliography

[Oppenheim1998] Oppenheim A. Questionnaire design, iterviewing and attitude measurement. . Pinter, 1998.

~~~

Matthew Loxton is a Knowledge Management professional and holds a Master’s degree in Knowledge Management from the University of Canberra. Mr. Loxton has extensive international experience and is currently available as a Knowledge Management consultant or as a permanent employee at an organization that wishes to put knowledge to work.

Advertisements

Tags: , , , , , , , , , , ,

3 Responses to “Aspects of Questionnaire Design”

  1. Rob Trenary Says:

    I really was enjoy reading your blog, great stuff. I’m glad that I went just here.

  2. Mari Pownell Says:

    I really was enjoy reading your blog, great stuff. I’m glad that I step in right here.

  3. Mitzi Cattell Says:

    Very good article. Coherent and informative just like a good article should be. I’ll definitively be back for more.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: