Many professions and fields offer certification exams. Generally speaking once someone takes the needed exams and receives certification in a given area the certification is only valid for a limited period of time. This is due to the fast pace at which things change. Almost every field of study has new information and tools added on a constant basis. Therefore as the capabilities needed to perform effectively in a field change it is necessary to update, revise and enhance the exams that certify people’s ability to perform specific roles. In today’s fast moving and fast changing environment certification exams are most often updated and revised because of product or service changes. In most cases a process similar to the development of the original exam is used to develop new versions of the exam. This is usually necessary because there are enough product changes to warrant a significant number of new exam questions. However, before discarding the old questions there is a lot of good data available which can inform the new development efforts.
Certification exam revision is best done following a structured approach. When updating an exam I recommend first identifying the items which are no longer valid due to changes in the product or service. Yes, that likely goes without saying. But, before you delete those items it may be worthwhile seeing how they performed. For those items that didn’t perform well you may be able to determine why they performed poorly and that information can be used to inform the new items that will be developed for the current update.
Even though certain questions may no longer be applicable to the new version of the product or service there is useful information available to the update effort. For example is it is likely that the existing exam contains several different types of questions on several different topics. There may be questions that ask about functionality or features or it’s compatibility to other products. There may be simple multiple-choice questions or perhaps there is a situation presented where there are several questions that reference that scenario. A first pass may look at how the various types of questions performed. What percentage of each type of question performed well and what percentage did not perform well? Is there a wide spread between how well different types of items performed? This is important because an individual’s score should be based on their ability to perform the activities associated with the job role as opposed to their ability to parse questions or understand the language of the test.
If you are working with an exam that addresses several topics for example services or competitive products, business problems addressed by product or services or sales strategies; then questions can be grouped by type and topic and reviewed for how well they performed. They can also be reviewed using individual aspects such as a specific service or product.
Statistical analyses can be carried out and you can use the resulting data to make decisions. However, what I want to address here is an analysis of how well the items performed based on the goals of the exam. For me the key questions to be answered are, first what did you want the people who receive certification to be able to do. This information is in the exam blueprint.
Reviewing how well test takers performed on each segment of the exam will provide information on how well the people who passed the exam are able to deliver to those areas covered by the exam. The reputation of the certification program rests not only on how many people achieve the certifications but also, and I would argue, more importantly, on how well the people who have the certification credentials are able to perform against the exam objectives.
The second question for me is how well test takers performed relative to section weights. On the heavily weighted sections is the performance evidenced at the level needed? Did people who received certification pass each section? If not, are there people who received certification but did not pass a highly weighted section. Did they receive the certification because they performed well on questions from the less relevant sections. If there are many people in this category consider if you need to change the section weights and have more questions address content from the key sections.
The purpose of a certification exam is to deliver the same questions to a wide audience of people within the field. Some may not be native speakers of the language of the exam. The item should assess whether the test taker has the knowledge needed to carry out the activity rather than the ability to understand complex language. Certification exams are designed to determine if the test taker can do those activities for which the certification is being awarded. Other tools or activities should be used to determine an individual’s ability to interact and comprehend a client’s communications.
I have heard people make a case for testing an individual’s ability to determine the problem or opportunity in more of a “real world” situation where a customer or client provides a great deal more information or perhaps incomplete information. My belief is that there is nothing wrong with testing for this competency, however, I believe these competency should be tested with another test, one which is designed to test comprehension, language, questioning etc. separate from a technical, product or industry specific certification instrument. Essentially if both the technical and comprehension factors are tested in the same item then one doesn’t know where remediation is needed. One also doesn’t know how to revise the item. Therefore, I recommend only testing for a single aspect in each item.
In looking at the items that didn’t perform well, consider if the language used in the stem (the questions itself) was simple and direct without additional information or ambiguous language. Some people may say that a complex or ambiguous question is okay. However, I don’t recommend using confusing or complex stems in certification exams for the reasons cited above.
Next I recommend looking at the items that are still valid. Determine how they are performing. Are revisions indicated for some or all of the response choices? Below are some steps to take before deleting items that are not performing effectively but cover content that still needs to be included on the exam.
- Show poorly performing items to experts, are they clearly phased
- Contact people performing the job role to learn the relationship of items that didn’t work to the work being done
- Do the questions that are performing poorly test real world activity
- How narrowly focused are the questions, do they address the work of the wide range of people taking the exam
- Do the poorly performing items follow recommended guidelines
See part 1 for additional exam development recommendations.
Doing the easier tasks first allows a determination of how much more needs to be done. Even with the best processes and procedures there are items that will not perform well. It is important when updating an exam to identify the cause of the poor performance of all the items that had bad results even if they are no longer relevant. A lot can be learned about how to develop new items even from items that are no longer technically relevant.
Once you have assessed the language and made any needed revisions I recommend you revise the answer options. Re-phase the correct response and develop new distractors. Test questions that are used for an extended period of time are often remembered by test takers and shared with others getting ready to take the exam. Once the total set of questions is developed it is recommended that all questions new or revised go through the same rigorous review process as was used for the initial development effort.
Summary
Exam development and exam updates are most effective when they follow a systematic process for development. When undertaking exam revision it is recommended that just as when the initial development effort was undertaken that a systems approach be employed. Identify the needed end results, validate that it is the result needed to achieve the desired outcomes, and then look at what is needed, what exists and what must be either developed or modified. It is important to keep the end result in mind as all the pieces of the development effort are carried out. As it is likely that there will be multiple exam questions developed by a group of exam writers, the test development manager will need to ensure that all aspects of the blueprint are covered by a sufficient number of questions.
Although you have more information as a result of doing an analysis of the results from the existing exam, you still need to ensure a sufficient number of items (new and revised) are developed and reviewed and approved so that items can be deleted as needed. A new practice test should also be developed. It is also advisable to have a few items available for interim updates that might be required after the exam is published.
In summary conducting a through analysis of the performance of the items on the existing exam will allow for more informed decision making as you develop the next version of the certification exam.
Systematic review and revision will lead to a credible and effective certification exam that will result in people who will effectively perform the activities defined in the exam objectives.
Tags: Assessment, Certification, Development, Exams, Instruction, Learning Theory, Systems Thinking, Test, Testing, Training
September 2, 2010 at 7:29 pm |
As someone who has considered getting a certification, I have a question; why do we need a certification for professionals? After all, if they are getting on the job training, they are moving with the flow of change and have demonstrated their abilities in real life situations. I believe that this is something of interest for both working and unemployed people.
September 3, 2010 at 12:22 pm |
Certifications are generally awarded after an individual has passed several exams. Once an individual has been certified, people who engage their services have a level of assurance that the individual who is performing a service to them will provide that service in an effective and efficient manner. Medical professionals are required to obtain multiple certifications. Today with the internet, it is very easy to use Google or another service to find out whether an individual is certified in many areas. Many computer companies offer certifications for their products and services. Service individuals in the computer industry who are certified can command a higher rate than those who are not certified.
A professional in the pharmaceutical industry told me that she promotes the competency of her personnel to third parties (FDA representatives and customers) by advertising her group members’ certifications. By obtaining certifications customers, clients and other members of their team know that an individual is current and up-to-date on the latest advances in their area of specialization.
We live in a world of constant innovation and change. There is considerable value in maintaining a professional edge. Certifications communicate that a professional has met the necessary requirements to perform the professional services they offer. In general, because of new knowledge and innovative updates in products and services, certifications are awarded for a limited time and often for a specific set of competencies. In the computer industry certifications are most often given for specific versions of products and services.
Certifications are similar to taking advanced training or coursework in a chosen field. However, certifications are more narrowly focused and define specific capabilities and knowledge whereas university coursework generally prepares one to apply what they have learned in many different contexts.
Tradespeople are licensed at different levels. When hiring a plumber or electrician one can ask whether they are licensed. Many tradespeople display their license numbers on their trucks. In the same way many technical people wear badges which indicate that they are certified in a specific area.
The use of certifications has grown significantly in recent years and will likely continue to be used to communicate knowledge and skills in given professional areas.