Developing Certification Exams

There are many factors to consider when implementing a certification exam program. The benefits of a well-developed and implemented program are many, ranging from well-prepared employees to increased profits due to a knowledgeable team. The tests themselves are most often a revenue generating process as many different populations can seek to be certified. But getting there requires multiple steps to ensure a quality exam.

It is critical to ensure that each step in the development process is well planned and executed. Below is an overview of steps that need to be taken to ensure a quality outcome. There are other steps that are followed by different development programs. However these are the major steps and processes.

• Begin by identifying the end results needed. What do the people who receive the certification need to be able to do.
• Once the end results have been identified, the steps needed to produce the end results have to be documented. For example, if the test covers driving different types of vehicles (perhaps cars, trucks, motorcycles, golf carts or others) include all the steps needed to drive each type of vehicle. This is known as a task analysis.
• Next a blueprint is developed. A team of experts most often develops the blueprint. The experts who develop the blueprint generally include practitioners and others who have expertise in the area. The blueprint describes each component of the task analysis. The blueprint identifies each area that will be tested by the exam. Most blueprints include the number of items for each objective that will be included on an exam. They also usually assign a weight to each topic area. Multiple examples of templates are available on line. Here is just one example.

https://www.nursingworld.org/~4acd24/globalassets/certification/certification-specialty-pages/resources/test-content-outlines/familynp-tco.pdf

• Once the blueprint is developed it needs to be validated by other subject matter experts (SMEs). Note that it is not expected that one person will have expertise in all areas covered by a full exam.
• At this point the item writing process begins. The very first step is to agree on the blueprint. Most often there are changes. Additions and some deletions can also happen, as topics may not be appropriate for the specific exam, may be covered on another exam, or might make the exam too long. Also there may be changes to the percentages and other edits before the team agrees on who will develop which items and the number of items each person will write.
• Handouts for the types of items to be included on the exam (for example multiple-choice, matching), how to write items, and what should and should not be included in an item are generally passed out and reviewed before participants begin writing the items. There are often handouts to explain how to input questions into the item development program used for item input.
• Here is an excellent set of guidelines for exam writing

http://www.cedma.org/blog/certification-exam-item-writing-best-practices

• It is important to let item writers develop just a few items (about three) and stop so that their items can be reviewed for feedback. Generally letting them write for no more than 30 minutes works well. It is also a good idea to review with the full team so that everyone hears the feedback. For a large group, the exam facilitator may want to just review one item written by each developer. Have them post their items in a way that allows for only the facilitator to see them. The exam facilitator should choose items to review so that different areas that need attention can be discussed.
• Once the first review is completed it is time to let the writers write more items. The exam facilitator needs to continue to review the developers’ items so that if they are making errors the facilitator can let them know what needs to be fixed. Usually a recommended rewrite helps. If the facilitator doesn’t know the subject area, they can use a gibberish word that will allow for the needed concept to be explained.
• After a period of time (usually two hours works) take a break and then begin to review the items. The facilitator can provide feedback on test development techniques and the developers need to provide feedback on the content of the questions. If there is agreement that the question is good, then it can be approved. If there isn’t agreement, the team can try to repair it. But in some cases it is better to delete the question, as time is often a factor in getting the number of questions that are needed and approved completed in the allotted timeframe.

Once the development workshop is completed the questions are sent out for review. Each question needs to be reviewed by multiple reviewers. Also each reviewer should only review a limited number of items to ensure the confidentiality of the exam.

That’s all for now. A future blog will cover additional information about certification exam development. Please let me know if you have comments or questions.

Writing Certification Exam Questions

This blog talks about some of what goes into the development of a certification exam. It is not intended to cover every aspect of exam development. If you have specific questions or want more detail about certification exam development, let me know and I might cover your questions in a future blog. There is much more that can be said and I plan to present additional blogs on this topic.

If you have expertise in an area where a certification exam is planned, you may be asked to participate in the development of the blueprint or exam questions. Blueprints are developed to guide the development of the exam questions. In general, the people who participate in the blueprint development do not also participate in the development of the exam questions based on that blueprint. Participants in the item development process use the blueprint to guide the development of questions. In addition to item developers, people knowledgeable in an area are also invited to review questions. Reviewers are assigned a subset of questions and asked to provide feedback. Generally several people are asked to review each of the questions.

The blueprint developers determine how much of the test will be devoted to each topic and the question developers review and revise as necessary. Generally speaking the item developers review topic areas and validate the assigned percentages. But, these are guidelines. It may be decided during item development that more or fewer questions on a topic are needed.

While there are many ways to develop certification exams, the key goal is to be able to certify that someone who has completed the exams for a specific area can be certified by the exam organization to be proficient in that area.

In general, certification exam item development processes require sufficient questions for multiple versions of the exam and also a practice test. The practice test is generally made available for review prior to taking the actual exam.

Most people who are involved in the development of exam questions have expertise in the subject areas covered by the exam. Very few of them have expertise in learning or exam development. Good question development is based on science. The science of how people learn and the application of that learning is an important element of exam development. This leads to better outcomes for both the exam takers and the organizations that might employ them.

In developing test questions it is important to consider what people will need to be able to do on the job. Too often questions ask for information that is a small component of the larger needed area. For example a question might ask which tools a service tech should have in their toolbox. However, knowing which tools are needed doesn’t indicate whether or not the test taker knows how to use those tools. There would need to be additional questions about the use of the tools.

One of the best ways to test the application of knowledge is to present a scenario that describes a situation. Then ask a series of questions that test the concept or skill needed to perform the work. Often the test delivery tool is limited to multiple-choice questions. However, there are ways to test complex activities or concepts using multiple-choice questions. For example a question can be posed about the order in which steps need to be taken.

Example: A tech needs to determine why a product isn’t working properly. What steps does the tech need to take and in which order?

Steps: (Which 4 steps are needed and in which order)
1
2
3
4
5
6
Answer choices:
A. 1,2,3,4
B. 3,1,5,2
C. 4,5,6,1
D. 6,4,2,1

A small scenario can be described and several questions about the scenario can be developed.

For example: Jane and Joe are driving on a road in an unfamiliar area. They need to get to a destination by a certain time to meet with the manager to discuss the services their company offers. They encounter a detour that is going to add time to their trip. What steps do you recommend they take and why?

Steps: (Which 4 steps do you recommend)
1
2
3
4
5
6
Answer choices:
A. 1,2,3,4
B. 3,1,5,2
C. 4,5,6,1
D. 6,4,2,1
Why do you recommend these steps:
A.
B.
C.
D.

Regardless of the exam delivery tool, you can – and should – develop items that ask about end results. Some delivery methods provide more question format options than others, but by keeping the necessary end results in mind, you can develop questions that will test what is required on-the-job.

Time for Change

Learning event development is changing at a very rapid pace. The year 2017 brings lots of forecasts for how things will change as well as reports on how things have already changed in the design and delivery of learning events.

https://www.forbes.com/sites/paycom/2017/02/14/learning-management-systems-101-rethinking-your-approach-to-employee-training/#68ee2d2d755b

A search for similar articles about changes to certification exam processes does not bring up similar types of posts.

The question that comes to mind is should we, as learning professionals, also look at whether changes need to be made to certification exams? Do we need to apply new concepts to what the exams cover? I say the answer is yes.

Many exams test minute components of a field in the belief that this will ensure the test taker will be able to perform in the job role. However, this is a concept that needs to be explored further. Exam teams write many test items to ensure they have a sufficient number of questions to create multiple forms of an exam. Multiple items also provide replacement items for questions that don’t perform well. Or for questions that become obsolete due to changes in products or processes. Items that test small components are also easier to develop and validate. But if we as learning professionals want to produce better job performance, we need to step up our game. We must help SMEs develop better test questions. We need to test and certify that the taker can perform the needed activities in the job role.

Without question, certification exams need to demonstrate that the exam is both valid and reliable. However, even if it is shown that items are both reliable and valid, if they cover just a small piece of an action, concept, or other component of the job role, how good is this information? Unless the test taker is able to integrate the pieces into the whole, there isn’t proof that correctly responding to such items demonstrates that the test taker can perform the needed end result.

Perhaps it is time to consider testing for the end results actually needed to be successful in a job role rather than testing the component parts of those activities. Too often we don’t test for the equivalent of driving a car but rather we test for the equivalent of whether the test taker knows where the brakes are, what road signs mean, and how to recover from a skid. All these activities are without a doubt extremely important activities for successfully driving a car. But it is time to test for end results. For example, asking a test taker to respond to questions related to something like in driving from point A to point B someone unexpectedly pulls out into your lane, what do you do? List response options that include multiple steps. Or list multiple steps in a column and ask the test taker to select all that apply.

As certification exam managers, we need to help item writers write meaningful certification exam questions. Item developers need to be able to write items that are close to essay items rather than asking about minute details in order to write enough items to produce multiple forms. Too often item developers write items, which are both valid, and reliable but do not assess whether the test taker can pull all the component parts together to solve problems. For example, correctly performing calculations is not enough. People need to be able to pull the component parts together and correctly apply them to a problem.

Another question that needs to be addressed is, do professionals need to memorize every facet of the work related to their field? Is it perhaps time to only require memorization of key concepts that must be second nature to professionals on the job? Can we use other methods to increase the number of items that are presented to the test taker to ensure valid and reliable results?

These are the questions that I would like to see explored in the literature. Can we perhaps design tests that include scenarios that present opportunities or problems that a professional is likely to see while performing the job role? For example, does a lawyer working on a case need to know how to prepare a defense for a client? Clearly not, they need to know which resources and documents they need to access. They also need to know how to use the tools available and how to compile an effective plan to defend their client. A financial planner also doesn’t need to know all the products available for their various clients. But they do need to know which types of financial products are appropriate for each segment of the population they serve. They also need to take the client’s level of openness to risk and other factors into consideration. These types of examples can be applied to every professional field.

I encourage learning professionals to explore and publish their findings about how we can improve the applicability of certification exam questions to job roles.

Blueprint for Certification, Part Two: Don’t throw the baby out with the bathwater

Many professions and fields offer certification exams. Generally speaking once someone takes the needed exams and receives certification in a given area the certification is only valid for a limited period of time. This is due to the fast pace at which things change. Almost every field of study has new information and tools added on a constant basis. Therefore as the capabilities needed to perform effectively in a field change it is necessary to update, revise and enhance the exams that certify people’s ability to perform specific roles. In today’s fast moving and fast changing environment certification exams are most often updated and revised because of product or service changes. In most cases a process similar to the development of the original exam is used to develop new versions of the exam. This is usually necessary because there are enough product changes to warrant a significant number of new exam questions. However, before discarding the old questions there is a lot of good data available which can inform the new development efforts.

Certification exam revision is best done following a structured approach. When updating an exam I recommend first identifying the items which are no longer valid due to changes in the product or service. Yes, that likely goes without saying. But, before you delete those items it may be worthwhile seeing how they performed. For those items that didn’t perform well you may be able to determine why they performed poorly and that information can be used to inform the new items that will be developed for the current update.

Even though certain questions may no longer be applicable to the new version of the product or service there is useful information available to the update effort. For example is it is likely that the existing exam contains several different types of questions on several different topics.  There may be questions that ask about functionality or features or it’s compatibility to other products. There may be simple multiple-choice questions or perhaps there is a situation presented where there are several questions that reference that scenario. A first pass may look at how the various types of questions performed. What percentage of each type of question performed well and what percentage did not perform well? Is there a wide spread between how well different types of items performed? This is important because an individual’s score should be based on their ability to perform the activities associated with the job role as opposed to their ability to parse questions or understand the language of the test.

If you are working with an exam that addresses several topics for example services or competitive products, business problems addressed by product or services or sales strategies; then questions can be grouped by type and topic and reviewed for how well they performed. They can also be reviewed using individual aspects such as a specific service or product.

Statistical analyses can be carried out and you can use the resulting data to make decisions. However, what I want to address here is an analysis of how well the items performed based on the goals of the exam. For me the key questions to be answered are, first what did you want the people who receive certification to be able to do. This information is in the exam blueprint.

Reviewing how well test takers performed on each segment of the exam will provide information on how well the people who passed the exam are able to deliver to those areas covered by the exam. The reputation of the certification program rests not only on how many people achieve the certifications but also, and I would argue, more importantly, on how well the people who have the certification credentials are able to perform against the exam objectives.

The second question for me is how well test takers performed relative to section weights. On the heavily weighted sections is the performance evidenced at the level needed? Did people who received certification pass each section? If not, are there people who received certification but did not pass a highly weighted section. Did they receive the certification because they performed well on questions from the less relevant sections. If there are many people in this category consider if you need to change the section weights and have more questions address content from the key sections.

The purpose of a certification exam is to deliver the same questions to a wide audience of people within the field. Some may not be native speakers of the language of the exam. The item should assess whether the test taker has the knowledge needed to carry out the activity rather than the ability to understand complex language. Certification exams are designed to determine if the test taker can do those activities for which the certification is being awarded. Other tools or activities should be used to determine an individual’s ability to interact and comprehend a client’s communications.

I have heard people make a case for testing an individual’s ability to determine the problem or opportunity in more of a “real world” situation where a customer or client provides a great deal more information or perhaps incomplete information. My belief is that there is nothing wrong with testing for this competency, however, I believe these competency should be tested with another test, one which is designed to test comprehension, language, questioning etc. separate from a technical, product or industry specific certification instrument. Essentially if both the technical and comprehension factors are tested in the same item then one doesn’t know where remediation is needed. One also doesn’t know how to revise the item. Therefore, I recommend only testing for a single aspect in each item.

In looking at the items that didn’t perform well, consider if the language used in the stem (the questions itself) was simple and direct without additional information or ambiguous language. Some people may say that a complex or ambiguous question is okay.   However, I don’t recommend using confusing or complex stems in certification exams for the reasons cited above.

Next I recommend looking at the items that are still valid. Determine how they are performing. Are revisions indicated for some or all of the response choices?  Below are some steps to take before deleting items that are not performing effectively but cover content that still needs to be included on the exam.

• Show poorly performing items to experts, are they clearly phased
• Contact people performing the job role to learn the relationship of items that didn’t work to the work being done
• Do the questions that are performing poorly test real world activity
• How narrowly focused are the questions, do they address the work of the wide range of people taking the exam
• Do the poorly performing items follow recommended guidelines

See part 1 for additional exam development recommendations.

Doing the easier tasks first allows a determination of how much more needs to be done. Even with the best processes and procedures there are items that will not perform well. It is important when updating an exam to identify the cause of the poor performance of all the items that had bad results even if they are no longer relevant. A lot can be learned about how to develop new items even from items that are no longer technically relevant.

Once you have assessed the language and made any needed revisions I recommend you revise the answer options. Re-phase the correct response and develop new distractors. Test questions that are used for an extended period of time are often remembered by test takers and shared with others getting ready to take the exam. Once the total set of questions is developed it is recommended that all questions new or revised go through the same rigorous review process as was used for the initial development effort.

Summary

Exam development and exam updates are most effective when they follow a systematic process for development. When undertaking exam revision it is recommended that just as when the initial development effort was undertaken that a systems approach be employed. Identify the needed end results, validate that it is the result needed to achieve the desired outcomes, and then look at what is needed, what exists and what must be either developed or modified. It is important to keep the end result in mind as all the pieces of the development effort are carried out. As it is likely that there will be multiple exam questions developed by a group of exam writers, the test development manager will need to ensure that all aspects of the blueprint are covered by a sufficient number of questions.

Although you have more information as a result of doing an analysis of the results from the existing exam, you still need to ensure a sufficient number of items (new and revised) are developed and reviewed and approved so that items can be deleted as needed. A new practice test should also be developed. It is also advisable to have a few items available for interim updates that might be required after the exam is published.

In summary conducting a through analysis of the performance of the items on the existing exam will allow for more informed decision making as you develop the next version of the certification exam.
Systematic review and revision will lead to a credible and effective certification exam that will result in people who will effectively perform the activities defined in the exam objectives.

Blueprint for Certification, Part One: Not All Certifications Are Created Equal

Today there are many corporations and professional organizations that offer certification programs.  Many companies and people in various industries seek these certifications.

Certification programs allow an entity to market expertise that certified individuals deliver when you interact with them in their professional capacities.  For example, knowing that a medical professional is certified indicates that the individual has shown the expertise covered by the exam.

Prior to the advent of certification exams, there wasn’t a standard set of criteria to judge a service provider’s expertise.  Word-of-mouth and direct experience with individuals was essentially how such decisions were made.  Fortunately, there are now many different certification programs in place.  For example, the Healthcare Quality Certification Board (HQCB) began publishing exams for healthcare professions in 1984.  The Certified Professional Healthcare Quality (CPHQ) certification has met with great success.

“Since the first examination was administered by the HQCB in 1984, more than 15,000 professionals from a wide variety of educational and employment backgrounds have registered for the CPHQ examination, with more than 11,000 achieving certified status.  There are currently over 6,800 active CPHQs in the United States and worldwide.”  http://www.cphq.org/

Certification programs also are advantageous to the organizations that offer them.  The time and money invested is most often recouped at a profit.  Not only are these organizations able to charge for the exams, they are also able to develop and deliver the instruction related to the exam topics.  All parties benefit from certification programs when they are done right.  Furthermore, if your enterprise has competitors who offer similar products or services, you can improve the chances that your organization’s products and services will be selected if users are certified in how to use your offerings.

However, not all certification programs are the same, some are better than others.  A certification exam should show that an individual is able to perform certain activities.  Those activities should be those needed to solve a problem or address an opportunity likely to be encountered on-the-job.  However, on occasion, questions on certification exams ask for some or even one of the steps, rather than testing for knowledge needed to solve the problem or take advantage of the opportunity.

Sometimes exams contain trick questions.  Questions, which are intentionally written to confuse the test taker are not an effective evaluation of the test takers ability to do the activity. Trick questions only test the ability to take a test or to parse a sentence.  This is a particularly poor design for non-native English speakers.  It is important to keep the purpose of a certification exam in mind when developing or accepting items for inclusion on an exam.  Certification exams certify that those who pass the exam can perform the tasks that are defined as being covered by the exam.

Test items should cover content an individual needs to know and do on the job.  Sometimes items that have nothing to do with performing the role might also be tested.  For example if the individual needs to drive a car from point A to point B asking questions about color of the interior of the car would not be relevant to the needed end result.  However, questions about the rules of the road would be appropriate.

Generally, certification tests are developed with enough questions to populate several forms of the test.  Test developers who are concerned with developing a sufficient number of test items sometimes write questions which test minor facts or information that normally would be looked up on the job.  These types of questions are unfair to all parties.  Test takers who are knowledgeable in the needed areas might not pass a poorly constructed exam that contains questions that cover small components of an overall process or activity.  For example, one doesn’t need to test on the location of the gas tank on a specific car.  Rather than testing to see if someone has memorized the location, it would be better to ask if they know about an arrow on the fuel gauge which points to the location of the gas tank.

My perspective is that a certification exam needs to be designed to evaluate the test takers ability to perform certain end results.  Test questions on a small piece of the overall process are often easier to develop but don’t necessarily test the ability to do what is needed.

Just like other technical fields, there are procedures and best practices for the development of test questions.  There are also conventions that lead to the development of items that are both valid and reliable.

Reliability and Validity

Measurement experts (and many educators) believe that every measurement device should possess certain qualities. Perhaps the two most common technical concepts in measurement are reliability and validity. Any kind of assessment, … must be developed in a way that gives the assessor accurate information about the performance of the individual. At one extreme, we wouldn’t have an individual paint a picture if we wanted to assess writing skills.

A. Reliability: Definition

• The degree of consistency between two measures of the same thing. (Mehrens and Lehman, 1987).

• The measure of how stable, dependable, trustworthy, and consistent a test is in measuring the same thing each time (Worthen et al., 1993)

For example, if we wish to measure a person’s weight, we would hope that the scale would register the same measure each time the person stepped on the scale.

…

B. Validity

1. Definition:

• Truthfulness: Does the test measure what it purports to measure? the extent to which certain inferences can be made from test scores or other measurement. (Mehrens and Lehman, 1987)

• The degree to which they accomplish the purpose for which they are being used. (Worthen et al., 1993)

For a test to be valid, or truthful, it must first be reliable. If we cannot even get a bathroom scale to give us a consistent weight measure, we certainly cannot expect it to be accurate. Note, however, that a measure might be consistent (reliable) but not accurate (valid). A scale may record weights as two pounds too heavy each time. In other words, reliability is a necessary but insufficient condition for validity. (Neither validity nor reliability is an either/or dichotomy; there are degrees of each.)

http://course1.winona.edu/lgray/el626/MandEtext3.html

Most good certification exams begin with a blueprint.  The blueprint defines the topics and sub-topics that are likely to be included on the exam.  In some cases, a blueprint will be developed by a group of subject matter experts who are familiar with the knowledge and skills needed to perform the job.  In some cases these are the experts who will also develop the test items.  In other cases, a different group of subject matter experts will develop the questions.  Blueprint development can also be based on the learning objectives from course offerings that cover exam topics.  Once a blueprint is developed, it should be put out for review by subject matter experts.  Most often these experts are performing the job roles and they are knowledgeable about what is needed to successfully carry out the needed activities.

The blueprint areas are usually weighted for importance and an associated number of items to be written are assigned to the topics to be covered.  Publishing the blueprint and associated sample test items is usually done so that the candidates can prepare for the exam.  Sample items for well-constructed exams are of the same difficulty and quality level as items the exam candidate might expect to find on the exam.

While processes vary and item development might be in workshop format, or submission by subject matter experts or another process, the end result is a certification exam that will allow individuals who pass the exam to present themselves as having the expertise that the exam certified.

Certification exams are developed and taken so that there is a standard.  People who have passed the exam can say they have met this standard.  It allows for selection of providers who have a credential that informs the “buyer” that they have engaged a professional for a role in which they have shown their expertise by passing an exam.  An exam that has been put in place to test many people who want to establish they have expertise in a specific area.

Certifications are valuable and worthwhile.  But a certification is only as good as the exam that produced it.  Avoid these common pit falls and stay focused on the goal: an exam that tests the skills, behavior and knowledge needed to carry out the tasks of the role.  In the next installment, I will discuss recommendations for updating and revising certification exams.

What Makes for a Good Certification Test?

Most certification tests are developed by teaming subject matter experts (SMEs) and certification test development experts. People responsible for writing the test questions need to have expertise in the subject area. They also need to know how to develop tests or have someone on the test writing team who knows how to develop credible, valid and reliable test items. A group or individual with expertise in testing, rather than the subject matter, usually provides this knowledge. Even so, certification tests don’t always live up to the standards set.

Tests are announced along with a description of what they will cover. The test development team has to make certain that the examination tests what the description promises. That may sound obvious, but the challenge lies in determining how to develop test questions which measure an individual’s ability to perform activities that they will need to carry out on-the-job.

The test writers need to develop questions and correct answers directly related to the skills being certified. They also need to write plausible incorrect answer choices. This is not as easy as it might seem. It is especially difficult when the test writers need to populate multiple versions of an exam. Additionally, extra items need to be produced, as some items will be deleted during the review process.

Developers will often concentrate on plausible incorrect responses. The resulting exam may get high marks for how well the items are constructed. But good construction is not enough. The question must also test the work that will be done. Take, for example, a question that asks what color a wire is and then lists four color choices. This may be a well constructed question when looking strictly at writing answer responses that are similar. But, knowing the color of a wire doesn’t test the ability to apply information in a work situation. If the question was about which wire to cut when defusing a bomb, then the question is testing application of knowledge. But it also isn’t sufficient to develop a question that asks the test taker to select the correct response. The answer choices must also be plausible. If wires are never pink, pink shouldn’t be an option.

A blueprint is used to build a home. Following a blueprint ensures that the home is built according to the original plan. Certification tests are also built according to a plan, also often called a blueprint. The blueprint provides detailed information for construction. The blueprint describes what should be included on the exam. There are multiple ways to develop an exam blueprint, but that is a topic for another blog. An exam development workshop generally begins with a review of the exam blueprint. The participants review, edit and modify the initial blueprint. As the workshop progresses and items are developed, the blueprint is reviewed, to ensure it is still valid. In some cases, content is modified or deleted from the blueprint. In other cases, content is added. The blueprint is normally not finalized until the test items have been reviewed, revised, edited and finalized. The blueprint is finalized only when the exam is ready for publication.

A Recommended Approach to Test Item Development

• Each workshop participant provides a brief summary of a situation they experienced that relates to one or more of the blueprint items (exam objective). Each blueprint item should have at least one situation associated with it. In some cases, a situation will cover more than one objective. This is not a problem; it might even lead to test questions which require higher level thought processes.
• Participants write a question based on one of the summary statements. At this point, they only write the question. Once the question has been reviewed and approved, they will develop the correct response and the distracters (the plausible incorrect responses).
• The exam facilitator and the workshop group review each question and critique it, both for relevance to the end result and for effective exam question development.
• Review and modify the answer choices as needed for content and effective test design.
• The exam facilitator may want the participants to develop another question following the above process.
• The facilitator determines which participants are ready to write additional questions without immediate review and feedback. It is recommended that these people write for approximately one hour and then a review and feedback session be conducted.
• Have the participants who still need a bit more support write about three more questions and response options. The facilitator should review and provide feedback to each of the three questions before the individual develops another question. If a question needs significant revision, the facilitator should provide feedback and then the participant should revise their own questions.
• The facilitator should review questions before presentation to the workshop participants. Individual feedback will help participants develop better questions. It will also move the process along without embarrassing any participants. Writing test questions isn’t easy. If the facilitator supports the people who are having problems one-on-one, it will lead to a better experience for all involved.
• During item review, the facilitator should remind the workshop team that the items should test activities that a certified support person will be doing. Participants may want to write items that test facts or information that would normally be looked up or found without memorization. This is not good test items development. Because so many items are needed, participants may push for including questions that test components. One option for effective test content is writing items which test finding the information rather than their ability to memorize. The team should ask if it is something done on the job. If it is, discuss and describe how and why it is done. This method will likely come up with a higher level capability and a better test item.

Comments in Blogs About Certification Exams
Below are some typical comments posted about certification exams.

“… a lot of questions that were ambiguous (had more than one answer, or no good answer). I ended up reading a number of them out loud to myself, shaking my head.

But where the CS2 test was full of ridiculous questions such as “does the tool tip in the navigator panel say ‘magnify’ or ‘zoom’?”, this CS3 test at least did give the sense that they were trying to test actual knowledge of how to Use the program.”

InDesign CS3 Certification Exam News, March 18 2008, Steve Werner

Variants on this comment are often posted for other certification exams. To avoid this problem, continually refer back to the initial summary experiences. Make sure these real experiences are incorporated into the questions. Discuss the process and steps taken for each example situation and job requirement being tested. This will result in a question that asks for higher level thinking not memorized information that can be looked up on the job.

Approaches for Item Review
After the development workshop, a few simple activities can lead to a better SME review process. These steps will allow SMEs to concentrate on technical aspects of the questions. An editorial review of the language, especially if the exam is going to multiple geographies, will help the technical reviewer concentrate on content. If possible, arrange for feedback to a sub-set of the items before the beta review. Conduct one-on-one sessions, if possible, with SMEs who didn’t participate in the exam development process. Another good way to receive quality feedback is to limit the number of items a reviewer is asked to review.

General Tips and Rules of Thumb
Focus on the work that is done using the knowledge being tested. Write items which test the application of the knowledge to real world situations. Ensure items are written so that all populations who will be sitting for the exam will understand the question. No tricks questions or responses. Don’t use idiomatic phases. Items should evaluate an individual’s ability to perform the job role, rather than their test taking ability, or their facility with the language in which the exam is written. If questions require minute technical detail, or are unrelated to the job, or require memorization of facts that can be looked up; these questions should be removed from the item pool.

It is important to know how customers will use the products. Someone who is certified in an area does not need to know every application of the products. However, they should know how the knowledge of the product can be applied. Knowing how knowledge can be applied, will allow the certified person to successfully handle a situation that is new or different.