Many professions and fields offer certification exams. Generally speaking once someone takes the needed exams and receives certification in a given area the certification is only valid for a limited period of time. This is due to the fast pace at which things change. Almost every field of study has new information and tools added on a constant basis. Therefore as the capabilities needed to perform effectively in a field change it is necessary to update, revise and enhance the exams that certify people’s ability to perform specific roles. In today’s fast moving and fast changing environment certification exams are most often updated and revised because of product or service changes. In most cases a process similar to the development of the original exam is used to develop new versions of the exam. This is usually necessary because there are enough product changes to warrant a significant number of new exam questions. However, before discarding the old questions there is a lot of good data available which can inform the new development efforts.
Certification exam revision is best done following a structured approach. When updating an exam I recommend first identifying the items which are no longer valid due to changes in the product or service. Yes, that likely goes without saying. But, before you delete those items it may be worthwhile seeing how they performed. For those items that didn’t perform well you may be able to determine why they performed poorly and that information can be used to inform the new items that will be developed for the current update.
Even though certain questions may no longer be applicable to the new version of the product or service there is useful information available to the update effort. For example is it is likely that the existing exam contains several different types of questions on several different topics. There may be questions that ask about functionality or features or it’s compatibility to other products. There may be simple multiple-choice questions or perhaps there is a situation presented where there are several questions that reference that scenario. A first pass may look at how the various types of questions performed. What percentage of each type of question performed well and what percentage did not perform well? Is there a wide spread between how well different types of items performed? This is important because an individual’s score should be based on their ability to perform the activities associated with the job role as opposed to their ability to parse questions or understand the language of the test.
If you are working with an exam that addresses several topics for example services or competitive products, business problems addressed by product or services or sales strategies; then questions can be grouped by type and topic and reviewed for how well they performed. They can also be reviewed using individual aspects such as a specific service or product.
Statistical analyses can be carried out and you can use the resulting data to make decisions. However, what I want to address here is an analysis of how well the items performed based on the goals of the exam. For me the key questions to be answered are, first what did you want the people who receive certification to be able to do. This information is in the exam blueprint.
Reviewing how well test takers performed on each segment of the exam will provide information on how well the people who passed the exam are able to deliver to those areas covered by the exam. The reputation of the certification program rests not only on how many people achieve the certifications but also, and I would argue, more importantly, on how well the people who have the certification credentials are able to perform against the exam objectives.
The second question for me is how well test takers performed relative to section weights. On the heavily weighted sections is the performance evidenced at the level needed? Did people who received certification pass each section? If not, are there people who received certification but did not pass a highly weighted section. Did they receive the certification because they performed well on questions from the less relevant sections. If there are many people in this category consider if you need to change the section weights and have more questions address content from the key sections.
The purpose of a certification exam is to deliver the same questions to a wide audience of people within the field. Some may not be native speakers of the language of the exam. The item should assess whether the test taker has the knowledge needed to carry out the activity rather than the ability to understand complex language. Certification exams are designed to determine if the test taker can do those activities for which the certification is being awarded. Other tools or activities should be used to determine an individual’s ability to interact and comprehend a client’s communications.
I have heard people make a case for testing an individual’s ability to determine the problem or opportunity in more of a “real world” situation where a customer or client provides a great deal more information or perhaps incomplete information. My belief is that there is nothing wrong with testing for this competency, however, I believe these competency should be tested with another test, one which is designed to test comprehension, language, questioning etc. separate from a technical, product or industry specific certification instrument. Essentially if both the technical and comprehension factors are tested in the same item then one doesn’t know where remediation is needed. One also doesn’t know how to revise the item. Therefore, I recommend only testing for a single aspect in each item.
In looking at the items that didn’t perform well, consider if the language used in the stem (the questions itself) was simple and direct without additional information or ambiguous language. Some people may say that a complex or ambiguous question is okay. However, I don’t recommend using confusing or complex stems in certification exams for the reasons cited above.
Next I recommend looking at the items that are still valid. Determine how they are performing. Are revisions indicated for some or all of the response choices? Below are some steps to take before deleting items that are not performing effectively but cover content that still needs to be included on the exam.
- Show poorly performing items to experts, are they clearly phased
- Contact people performing the job role to learn the relationship of items that didn’t work to the work being done
- Do the questions that are performing poorly test real world activity
- How narrowly focused are the questions, do they address the work of the wide range of people taking the exam
- Do the poorly performing items follow recommended guidelines
See part 1 for additional exam development recommendations.
Doing the easier tasks first allows a determination of how much more needs to be done. Even with the best processes and procedures there are items that will not perform well. It is important when updating an exam to identify the cause of the poor performance of all the items that had bad results even if they are no longer relevant. A lot can be learned about how to develop new items even from items that are no longer technically relevant.
Once you have assessed the language and made any needed revisions I recommend you revise the answer options. Re-phase the correct response and develop new distractors. Test questions that are used for an extended period of time are often remembered by test takers and shared with others getting ready to take the exam. Once the total set of questions is developed it is recommended that all questions new or revised go through the same rigorous review process as was used for the initial development effort.
Summary
Exam development and exam updates are most effective when they follow a systematic process for development. When undertaking exam revision it is recommended that just as when the initial development effort was undertaken that a systems approach be employed. Identify the needed end results, validate that it is the result needed to achieve the desired outcomes, and then look at what is needed, what exists and what must be either developed or modified. It is important to keep the end result in mind as all the pieces of the development effort are carried out. As it is likely that there will be multiple exam questions developed by a group of exam writers, the test development manager will need to ensure that all aspects of the blueprint are covered by a sufficient number of questions.
Although you have more information as a result of doing an analysis of the results from the existing exam, you still need to ensure a sufficient number of items (new and revised) are developed and reviewed and approved so that items can be deleted as needed. A new practice test should also be developed. It is also advisable to have a few items available for interim updates that might be required after the exam is published.
In summary conducting a through analysis of the performance of the items on the existing exam will allow for more informed decision making as you develop the next version of the certification exam.
Systematic review and revision will lead to a credible and effective certification exam that will result in people who will effectively perform the activities defined in the exam objectives.