The last few years have seen a general improvement in the quality of learning events offered for business learners. We have learned that chunking content, offering hands-on activities, and self-tests improve retention. Even more importantly it improves how well learners apply that content. While all of this is great, there are still some improvements to be made. I would like to address how we develop test questions for large populations and provide some recommendations for developing more complex items.
I have often heard course developers complain that because of the need to develop multiple-choice questions they are unable to develop questions that actually test what they believe the learners need to be able to do. Hopefully, this blog and other posts will begin to provide more information about how they can more effectively develop multiple-choice questions.
Developers are required to use multiple-choice questions in most cases because we don’t have the bandwidth to read and review items for large groups of test takers. Given that requirement, there are ways to improve our multiple-choice items. Multiple-choice questions don’t necessarily mean simple questions. There are numerous ways to ask multiple-choice questions that allow for developing higher order items.
Here I will go into a little bit of detail on a few methods that have a good success rate. As always, if you have any questions, please leave a comment below.
Pictures or graphics – If the test delivery tool allows for including graphics, they can be used in questions and/or response options. Show pictures of several items in each response choice, with only one answer choice showing all the correct matches.
Matching – ask a question that requires that the test taker match items in two columns. Include more items in the second column so that the match for the last item in column A doesn’t fall into place. Response options should list the letter (or number) of several items in each answer choice. Only one answer choice would list all the correct matches.
Example: Match the items in column A with the item in column B that is needed to operate that item. Note column B contains extra options.
| Column A | Column B
|
| A1. Clock with plug | B1. Electricity |
| A2. Car | B2. Gasoline |
| Etc. | B3. Oven |
| Etc.
|
Answer Options:
- A1 and B1, A2 and B2 etc.
- A1 and B2, A2 and B3 etc.
- XYZ and DEG, GXT and TOQ etc.
Putting a machine or other physical product together
If learners have been taught how to repair, fix or configure a machine a test question can ask in which order a machine should be put together. The answer options could show different parts that need to be installed. The question can ask the order in which to install the component parts.
Multiple Step Process – The first part of the question describes a situation, process or problem that requires several steps to resolve. The learner is asked for the steps and the order in which to complete them. A list of steps is then presented for the test taker to use to resolve the situation.
With this type of question it can be stated that there are many options that are not listed, explain that the test taker should only consider the options listed to respond to the question. Response options list the letter or number of several items in each answer choice; only one answer choice lists all the correct matches in the correct order.
Example: In which order should a person begin to plan for a long trip? There are many options that are not listed here, consider only the options listed to respond to this question.
- Determine the trip distance
- Decide on the mode of transportation
- Identify the costs of the different options available
- Identify the amount of time required for each option
- Find a friend to travel with to help defray the costs
- Consider all factors to find a good time to travel
Answer Options:
- 1,2,3,5
- 2,4,1,3
- 6,5,2,3
- 4,3,2,1
Provide a short description of a process – Describe a process and ask how it can be applied, or what can go wrong, or any other concept that the learner needs to know. Response options list the letter or number of several items in each answer choice. Only one answer choice lists all the correct matches to the content the learner needs to know.
Ordering – In this technique a question can list several steps in a process. The test taker is asked which steps need to be taken and in which order, to achieve a result. It is a good idea to list more steps than are needed. Mention in the body of the first part of the question that not all steps listed will be needed. Response options list the letter or number of several items in each answer choice; only one answer choice lists all the correct matches in the correct order.
Example: What steps need to be taken and in which order, to develop a KWPQ? NOTE: There are more steps listed than are need.
- do the thing
- list the thing
- put the thing in the place
- start the thing
- set the timer for the thing
- display the thing
- process the thing
Answer Options:
- 1,3,5,7
- 2,1,7,4
- 6,2,3,5
- 4,6,1,3
These are just a few ways and there’s a lot more to the craft of successfully developing tests with multiple-choice questions. But hopefully the above thoughts will help. As always, I’m glad to answer questions.
Blueprints for Learning 