Learning Styles

For several years learning professionals were writing and reading about different learning styles. The talk was about how different people learn more effectively from different styles of content presentation. In more recent years some educators are saying that learning styles are not a factor. They are saying that all learners can learn from good presentations regardless of the modality.

In a January 9, 2019 Inside Higher Ed article Greg Toppo, a senior editor at Inside Higher Ed., discusses the topic of learning styles.

https://www.insidehighered.com/news/2019/01/09/learning-styles-debate-its-instructors-vs-psychologists

“But Daniel Willingham, a cognitive psychologist and professor at the University of Virginia, said the categories themselves ‘haven’t been shown to mean anything.’ Nonetheless, recent surveys have found that about 90 percent of Virginia students believe in them.”

In this article Toppo also says there are learning professionals who very much support the concept of different learning styles.

“Richard Felder, a professor emeritus of chemical engineering at North Carolina State University who has written in support of learning styles, said psychologists have spent decades working to debunk the theory. ‘On the other side are literally millions of people who have used learning styles to design instruction’ and to help students become better learners, he said.”

A quick search of scholarly articles about learning styles produces a host of articles. In recent years there have been many learning professionals who have written to deride the concept of learning styles. At the same time there are many others who have written scholarly articles based on the theory that different people learn differently.

A good number of learning professionals continue to develop and deliver learning products designed to deliver instruction geared to different learning styles. I am not sure there is any harm in doing so, even if it is true that the learning styles theory is invalid. I also believe that the learning style theory is a comfortable concept. We believe we understand what needs to go into content for each type of learner. We can include graphics and pictures and so forth for the visual learner, audio for the auditory learner, and materials that can be handled for the kinesthetic learner.

However, I believe that a good learning professional, using whatever methodology works for them, will provide what is needed for learning to occur. It is the design of the learning product as well as the instructor’s delivery of the content that will provide the learner with what they need to grasp the content not the learner’s perceived learning modality.

I am perplexed at how much time and effort is being put into arguments for and against the concept of learning styles. I understand that the research that has been reported is that learning styles do not have merit. I think it is time for researchers to design some new studies. Learning events that make use of different modalities to deliver the same instruction should be developed. In addition instruction covering the same content using various modalities should also use different instructors’ preferred delivery styles.

Each of these instructional units should administer the same evaluation instruments to each group of learners. Comparisons of the outcomes should be informative and might help us to understand if there are indeed differences, or if all well designed and well delivered modalities lead to effective learning outcomes.

Writing Complex Multiple Choice Questions

The last few years have seen a general improvement in the quality of learning events offered for business learners. We have learned that chunking content, offering hands-on activities, and self-tests improve retention. Even more importantly it improves how well learners apply that content. While all of this is great, there are still some improvements to be made. I would like to address how we develop test questions for large populations and provide some recommendations for developing more complex items.

I have often heard course developers complain that because of the need to develop multiple-choice questions they are unable to develop questions that actually test what they believe the learners need to be able to do. Hopefully, this blog and other posts will begin to provide more information about how they can more effectively develop multiple-choice questions.

Developers are required to use multiple-choice questions in most cases because we don’t have the bandwidth to read and review items for large groups of test takers. Given that requirement, there are ways to improve our multiple-choice items. Multiple-choice questions don’t necessarily mean simple questions. There are numerous ways to ask multiple-choice questions that allow for developing higher order items. 

Here I will go into a little bit of detail on a few methods that have a good success rate. As always, if you have any questions, please leave a comment below.

Pictures or graphics – If the test delivery tool allows for including graphics, they can be used in questions and/or response options. Show pictures of several items in each response choice, with only one answer choice showing all the correct matches.

Matching – ask a question that requires that the test taker match items in two columns. Include more items in the second column so that the match for the last item in column A doesn’t fall into place. Response options should list the letter (or number) of several items in each answer choice. Only one answer choice would list all the correct matches.

Example: Match the items in column A with the item in column B that is needed to operate that item. Note column B contains extra options.

Column A	Column B
A1. Clock with plug	B1. Electricity
A2. Car	B2. Gasoline
Etc.	B3. Oven
	Etc.

Answer Options:

1. A1 and B1, A2 and B2 etc.
2. A1 and B2, A2 and B3 etc.
3. XYZ and DEG, GXT and TOQ etc.

Putting a machine or other physical product together

If learners have been taught how to repair, fix or configure a machine a test question can ask in which order a machine should be put together. The answer options could show different parts that need to be installed. The question can ask the order in which to install the component parts.

Multiple Step Process – The first part of the question describes a situation, process or problem that requires several steps to resolve. The learner is asked for the steps and the order in which to complete them. A list of steps is then presented for the test taker to use to resolve the situation.

With this type of question it can be stated that there are many options that are not listed, explain that the test taker should only consider the options listed to respond to the question. Response options list the letter or number of several items in each answer choice; only one answer choice lists all the correct matches in the correct order.

Example: In which order should a person begin to plan for a long trip? There are many options that are not listed here, consider only the options listed to respond to this question.

1. Determine the trip distance
2. Decide on the mode of transportation
3. Identify the costs of the different options available
4. Identify the amount of time required for each option
5. Find a friend to travel with to help defray the costs
6. Consider all factors to find a good time to travel

Answer Options:

1. 1,2,3,5
2. 2,4,1,3
3. 6,5,2,3
4. 4,3,2,1

Provide a short description of a process – Describe a process and ask how it can be applied, or what can go wrong, or any other concept that the learner needs to know. Response options list the letter or number of several items in each answer choice. Only one answer choice lists all the correct matches to the content the learner needs to know.

Ordering – In this technique a question can list several steps in a process. The test taker is asked which steps need to be taken and in which order, to achieve a result. It is a good idea to list more steps than are needed. Mention in the body of the first part of the question that not all steps listed will be needed. Response options list the letter or number of several items in each answer choice; only one answer choice lists all the correct matches in the correct order.

Example: What steps need to be taken and in which order, to develop a KWPQ? NOTE: There are more steps listed than are need.

1. do the thing
2. list the thing
3. put the thing in the place
4. start the thing
5. set the timer for the thing
6. display the thing
7. process the thing

Answer Options:

1. 1,3,5,7
2. 2,1,7,4
3. 6,2,3,5
4. 4,6,1,3

These are just a few ways and there’s a lot more to the craft of successfully developing tests with multiple-choice questions. But hopefully the above thoughts will help. As always, I’m glad to answer questions.

 

A To Do List for 2018

As we begin to look toward 2018, I recommend we look at both our successes and the areas that still need work. We need to look at how our learning products and processes have evolved and where we still need to improve in 2018.

Here are the areas I think we need to look at. Please respond to this blog with additional areas to add. We’ll take a closer look at successes next week and I will be blogging about these topics (and more!) as we venture into 2018.

What I Intend to Write About in 2018:

1. Explain new concepts, techniques, processes, or other new content
a. Why is it needed?
b. How does it integrate?
c. How does it do what it does?

2. Practice
a. Identifying effective exercises
b. Developing learn-by-doing practice items
c. Develop practice items that require learner interaction

3. Feedback
a. How to develop feedback that works
b. What type of feedback doesn’t work and why
c. Where and when to use feedback in different types of learning events

4. Examples
a. Real world applications of what has been taught
b. Use stories to help learning occur
c. Use the right example at the right time

5. Integrating new tools into learning events
a. How tools support learning
b. Why new or fun doesn’t necessarily mean effective
c. Learning which tools will work for an audience

6. Using new techniques to develop learning events
a. How to evaluate what is new for different learning needs
b. How to validate what is new for different learning needs
c. Using formative evaluation techniques

7. Team Learning
a. What is Team Learning?
b. When is Team Learning applicable?
c. When does Team Learning work?

8. Integrate vs. References
a. Learning for memorization
b. Learning to apply to new situations
c. Using references

9. Keeping it simple and short
a. Mini modules
b. Just the facts
c. Test, test, test to find what is missing

10. Switch It Up
a. Keeping the learners engaged
b. Learner participation
c. Get learner feedback

Writing Certification Exam Questions

This blog talks about some of what goes into the development of a certification exam. It is not intended to cover every aspect of exam development. If you have specific questions or want more detail about certification exam development, let me know and I might cover your questions in a future blog. There is much more that can be said and I plan to present additional blogs on this topic.

If you have expertise in an area where a certification exam is planned, you may be asked to participate in the development of the blueprint or exam questions. Blueprints are developed to guide the development of the exam questions. In general, the people who participate in the blueprint development do not also participate in the development of the exam questions based on that blueprint. Participants in the item development process use the blueprint to guide the development of questions. In addition to item developers, people knowledgeable in an area are also invited to review questions. Reviewers are assigned a subset of questions and asked to provide feedback. Generally several people are asked to review each of the questions.

The blueprint developers determine how much of the test will be devoted to each topic and the question developers review and revise as necessary. Generally speaking the item developers review topic areas and validate the assigned percentages. But, these are guidelines. It may be decided during item development that more or fewer questions on a topic are needed.

While there are many ways to develop certification exams, the key goal is to be able to certify that someone who has completed the exams for a specific area can be certified by the exam organization to be proficient in that area.

In general, certification exam item development processes require sufficient questions for multiple versions of the exam and also a practice test. The practice test is generally made available for review prior to taking the actual exam.

Most people who are involved in the development of exam questions have expertise in the subject areas covered by the exam. Very few of them have expertise in learning or exam development. Good question development is based on science. The science of how people learn and the application of that learning is an important element of exam development. This leads to better outcomes for both the exam takers and the organizations that might employ them.

In developing test questions it is important to consider what people will need to be able to do on the job. Too often questions ask for information that is a small component of the larger needed area. For example a question might ask which tools a service tech should have in their toolbox. However, knowing which tools are needed doesn’t indicate whether or not the test taker knows how to use those tools. There would need to be additional questions about the use of the tools.

One of the best ways to test the application of knowledge is to present a scenario that describes a situation. Then ask a series of questions that test the concept or skill needed to perform the work. Often the test delivery tool is limited to multiple-choice questions. However, there are ways to test complex activities or concepts using multiple-choice questions. For example a question can be posed about the order in which steps need to be taken.

Example: A tech needs to determine why a product isn’t working properly. What steps does the tech need to take and in which order?

Steps: (Which 4 steps are needed and in which order)
1
2
3
4
5
6
Answer choices:
A. 1,2,3,4
B. 3,1,5,2
C. 4,5,6,1
D. 6,4,2,1

A small scenario can be described and several questions about the scenario can be developed.

For example: Jane and Joe are driving on a road in an unfamiliar area. They need to get to a destination by a certain time to meet with the manager to discuss the services their company offers. They encounter a detour that is going to add time to their trip. What steps do you recommend they take and why?

Steps: (Which 4 steps do you recommend)
1
2
3
4
5
6
Answer choices:
A. 1,2,3,4
B. 3,1,5,2
C. 4,5,6,1
D. 6,4,2,1
Why do you recommend these steps:
A.
B.
C.
D.

Regardless of the exam delivery tool, you can – and should – develop items that ask about end results. Some delivery methods provide more question format options than others, but by keeping the necessary end results in mind, you can develop questions that will test what is required on-the-job.

Time for Change

Learning event development is changing at a very rapid pace. The year 2017 brings lots of forecasts for how things will change as well as reports on how things have already changed in the design and delivery of learning events.

https://www.forbes.com/sites/paycom/2017/02/14/learning-management-systems-101-rethinking-your-approach-to-employee-training/#68ee2d2d755b

A search for similar articles about changes to certification exam processes does not bring up similar types of posts.

The question that comes to mind is should we, as learning professionals, also look at whether changes need to be made to certification exams? Do we need to apply new concepts to what the exams cover? I say the answer is yes.

Many exams test minute components of a field in the belief that this will ensure the test taker will be able to perform in the job role. However, this is a concept that needs to be explored further. Exam teams write many test items to ensure they have a sufficient number of questions to create multiple forms of an exam. Multiple items also provide replacement items for questions that don’t perform well. Or for questions that become obsolete due to changes in products or processes. Items that test small components are also easier to develop and validate. But if we as learning professionals want to produce better job performance, we need to step up our game. We must help SMEs develop better test questions. We need to test and certify that the taker can perform the needed activities in the job role.

Without question, certification exams need to demonstrate that the exam is both valid and reliable. However, even if it is shown that items are both reliable and valid, if they cover just a small piece of an action, concept, or other component of the job role, how good is this information? Unless the test taker is able to integrate the pieces into the whole, there isn’t proof that correctly responding to such items demonstrates that the test taker can perform the needed end result.

Perhaps it is time to consider testing for the end results actually needed to be successful in a job role rather than testing the component parts of those activities. Too often we don’t test for the equivalent of driving a car but rather we test for the equivalent of whether the test taker knows where the brakes are, what road signs mean, and how to recover from a skid. All these activities are without a doubt extremely important activities for successfully driving a car. But it is time to test for end results. For example, asking a test taker to respond to questions related to something like in driving from point A to point B someone unexpectedly pulls out into your lane, what do you do? List response options that include multiple steps. Or list multiple steps in a column and ask the test taker to select all that apply.

As certification exam managers, we need to help item writers write meaningful certification exam questions. Item developers need to be able to write items that are close to essay items rather than asking about minute details in order to write enough items to produce multiple forms. Too often item developers write items, which are both valid, and reliable but do not assess whether the test taker can pull all the component parts together to solve problems. For example, correctly performing calculations is not enough. People need to be able to pull the component parts together and correctly apply them to a problem.

Another question that needs to be addressed is, do professionals need to memorize every facet of the work related to their field? Is it perhaps time to only require memorization of key concepts that must be second nature to professionals on the job? Can we use other methods to increase the number of items that are presented to the test taker to ensure valid and reliable results?

These are the questions that I would like to see explored in the literature. Can we perhaps design tests that include scenarios that present opportunities or problems that a professional is likely to see while performing the job role? For example, does a lawyer working on a case need to know how to prepare a defense for a client? Clearly not, they need to know which resources and documents they need to access. They also need to know how to use the tools available and how to compile an effective plan to defend their client. A financial planner also doesn’t need to know all the products available for their various clients. But they do need to know which types of financial products are appropriate for each segment of the population they serve. They also need to take the client’s level of openness to risk and other factors into consideration. These types of examples can be applied to every professional field.

I encourage learning professionals to explore and publish their findings about how we can improve the applicability of certification exam questions to job roles.

Blueprint for Certification, Part Two: Don’t throw the baby out with the bathwater

Many professions and fields offer certification exams. Generally speaking once someone takes the needed exams and receives certification in a given area the certification is only valid for a limited period of time. This is due to the fast pace at which things change. Almost every field of study has new information and tools added on a constant basis. Therefore as the capabilities needed to perform effectively in a field change it is necessary to update, revise and enhance the exams that certify people’s ability to perform specific roles. In today’s fast moving and fast changing environment certification exams are most often updated and revised because of product or service changes. In most cases a process similar to the development of the original exam is used to develop new versions of the exam. This is usually necessary because there are enough product changes to warrant a significant number of new exam questions. However, before discarding the old questions there is a lot of good data available which can inform the new development efforts.

Certification exam revision is best done following a structured approach. When updating an exam I recommend first identifying the items which are no longer valid due to changes in the product or service. Yes, that likely goes without saying. But, before you delete those items it may be worthwhile seeing how they performed. For those items that didn’t perform well you may be able to determine why they performed poorly and that information can be used to inform the new items that will be developed for the current update.

Even though certain questions may no longer be applicable to the new version of the product or service there is useful information available to the update effort. For example is it is likely that the existing exam contains several different types of questions on several different topics.  There may be questions that ask about functionality or features or it’s compatibility to other products. There may be simple multiple-choice questions or perhaps there is a situation presented where there are several questions that reference that scenario. A first pass may look at how the various types of questions performed. What percentage of each type of question performed well and what percentage did not perform well? Is there a wide spread between how well different types of items performed? This is important because an individual’s score should be based on their ability to perform the activities associated with the job role as opposed to their ability to parse questions or understand the language of the test.

If you are working with an exam that addresses several topics for example services or competitive products, business problems addressed by product or services or sales strategies; then questions can be grouped by type and topic and reviewed for how well they performed. They can also be reviewed using individual aspects such as a specific service or product.

Statistical analyses can be carried out and you can use the resulting data to make decisions. However, what I want to address here is an analysis of how well the items performed based on the goals of the exam. For me the key questions to be answered are, first what did you want the people who receive certification to be able to do. This information is in the exam blueprint.

Reviewing how well test takers performed on each segment of the exam will provide information on how well the people who passed the exam are able to deliver to those areas covered by the exam. The reputation of the certification program rests not only on how many people achieve the certifications but also, and I would argue, more importantly, on how well the people who have the certification credentials are able to perform against the exam objectives.

The second question for me is how well test takers performed relative to section weights. On the heavily weighted sections is the performance evidenced at the level needed? Did people who received certification pass each section? If not, are there people who received certification but did not pass a highly weighted section. Did they receive the certification because they performed well on questions from the less relevant sections. If there are many people in this category consider if you need to change the section weights and have more questions address content from the key sections.

The purpose of a certification exam is to deliver the same questions to a wide audience of people within the field. Some may not be native speakers of the language of the exam. The item should assess whether the test taker has the knowledge needed to carry out the activity rather than the ability to understand complex language. Certification exams are designed to determine if the test taker can do those activities for which the certification is being awarded. Other tools or activities should be used to determine an individual’s ability to interact and comprehend a client’s communications.

I have heard people make a case for testing an individual’s ability to determine the problem or opportunity in more of a “real world” situation where a customer or client provides a great deal more information or perhaps incomplete information. My belief is that there is nothing wrong with testing for this competency, however, I believe these competency should be tested with another test, one which is designed to test comprehension, language, questioning etc. separate from a technical, product or industry specific certification instrument. Essentially if both the technical and comprehension factors are tested in the same item then one doesn’t know where remediation is needed. One also doesn’t know how to revise the item. Therefore, I recommend only testing for a single aspect in each item.

In looking at the items that didn’t perform well, consider if the language used in the stem (the questions itself) was simple and direct without additional information or ambiguous language. Some people may say that a complex or ambiguous question is okay.   However, I don’t recommend using confusing or complex stems in certification exams for the reasons cited above.

Next I recommend looking at the items that are still valid. Determine how they are performing. Are revisions indicated for some or all of the response choices?  Below are some steps to take before deleting items that are not performing effectively but cover content that still needs to be included on the exam.

• Show poorly performing items to experts, are they clearly phased
• Contact people performing the job role to learn the relationship of items that didn’t work to the work being done
• Do the questions that are performing poorly test real world activity
• How narrowly focused are the questions, do they address the work of the wide range of people taking the exam
• Do the poorly performing items follow recommended guidelines

See part 1 for additional exam development recommendations.

Doing the easier tasks first allows a determination of how much more needs to be done. Even with the best processes and procedures there are items that will not perform well. It is important when updating an exam to identify the cause of the poor performance of all the items that had bad results even if they are no longer relevant. A lot can be learned about how to develop new items even from items that are no longer technically relevant.

Once you have assessed the language and made any needed revisions I recommend you revise the answer options. Re-phase the correct response and develop new distractors. Test questions that are used for an extended period of time are often remembered by test takers and shared with others getting ready to take the exam. Once the total set of questions is developed it is recommended that all questions new or revised go through the same rigorous review process as was used for the initial development effort.

Summary

Exam development and exam updates are most effective when they follow a systematic process for development. When undertaking exam revision it is recommended that just as when the initial development effort was undertaken that a systems approach be employed. Identify the needed end results, validate that it is the result needed to achieve the desired outcomes, and then look at what is needed, what exists and what must be either developed or modified. It is important to keep the end result in mind as all the pieces of the development effort are carried out. As it is likely that there will be multiple exam questions developed by a group of exam writers, the test development manager will need to ensure that all aspects of the blueprint are covered by a sufficient number of questions.

Although you have more information as a result of doing an analysis of the results from the existing exam, you still need to ensure a sufficient number of items (new and revised) are developed and reviewed and approved so that items can be deleted as needed. A new practice test should also be developed. It is also advisable to have a few items available for interim updates that might be required after the exam is published.

In summary conducting a through analysis of the performance of the items on the existing exam will allow for more informed decision making as you develop the next version of the certification exam.
Systematic review and revision will lead to a credible and effective certification exam that will result in people who will effectively perform the activities defined in the exam objectives.

Blueprint for Certification, Part One: Not All Certifications Are Created Equal

Today there are many corporations and professional organizations that offer certification programs.  Many companies and people in various industries seek these certifications.

Certification programs allow an entity to market expertise that certified individuals deliver when you interact with them in their professional capacities.  For example, knowing that a medical professional is certified indicates that the individual has shown the expertise covered by the exam.

Prior to the advent of certification exams, there wasn’t a standard set of criteria to judge a service provider’s expertise.  Word-of-mouth and direct experience with individuals was essentially how such decisions were made.  Fortunately, there are now many different certification programs in place.  For example, the Healthcare Quality Certification Board (HQCB) began publishing exams for healthcare professions in 1984.  The Certified Professional Healthcare Quality (CPHQ) certification has met with great success.

“Since the first examination was administered by the HQCB in 1984, more than 15,000 professionals from a wide variety of educational and employment backgrounds have registered for the CPHQ examination, with more than 11,000 achieving certified status.  There are currently over 6,800 active CPHQs in the United States and worldwide.”  http://www.cphq.org/

Certification programs also are advantageous to the organizations that offer them.  The time and money invested is most often recouped at a profit.  Not only are these organizations able to charge for the exams, they are also able to develop and deliver the instruction related to the exam topics.  All parties benefit from certification programs when they are done right.  Furthermore, if your enterprise has competitors who offer similar products or services, you can improve the chances that your organization’s products and services will be selected if users are certified in how to use your offerings.

However, not all certification programs are the same, some are better than others.  A certification exam should show that an individual is able to perform certain activities.  Those activities should be those needed to solve a problem or address an opportunity likely to be encountered on-the-job.  However, on occasion, questions on certification exams ask for some or even one of the steps, rather than testing for knowledge needed to solve the problem or take advantage of the opportunity.

Sometimes exams contain trick questions.  Questions, which are intentionally written to confuse the test taker are not an effective evaluation of the test takers ability to do the activity. Trick questions only test the ability to take a test or to parse a sentence.  This is a particularly poor design for non-native English speakers.  It is important to keep the purpose of a certification exam in mind when developing or accepting items for inclusion on an exam.  Certification exams certify that those who pass the exam can perform the tasks that are defined as being covered by the exam.

Test items should cover content an individual needs to know and do on the job.  Sometimes items that have nothing to do with performing the role might also be tested.  For example if the individual needs to drive a car from point A to point B asking questions about color of the interior of the car would not be relevant to the needed end result.  However, questions about the rules of the road would be appropriate.

Generally, certification tests are developed with enough questions to populate several forms of the test.  Test developers who are concerned with developing a sufficient number of test items sometimes write questions which test minor facts or information that normally would be looked up on the job.  These types of questions are unfair to all parties.  Test takers who are knowledgeable in the needed areas might not pass a poorly constructed exam that contains questions that cover small components of an overall process or activity.  For example, one doesn’t need to test on the location of the gas tank on a specific car.  Rather than testing to see if someone has memorized the location, it would be better to ask if they know about an arrow on the fuel gauge which points to the location of the gas tank.

My perspective is that a certification exam needs to be designed to evaluate the test takers ability to perform certain end results.  Test questions on a small piece of the overall process are often easier to develop but don’t necessarily test the ability to do what is needed.

Just like other technical fields, there are procedures and best practices for the development of test questions.  There are also conventions that lead to the development of items that are both valid and reliable.

Reliability and Validity

Measurement experts (and many educators) believe that every measurement device should possess certain qualities. Perhaps the two most common technical concepts in measurement are reliability and validity. Any kind of assessment, … must be developed in a way that gives the assessor accurate information about the performance of the individual. At one extreme, we wouldn’t have an individual paint a picture if we wanted to assess writing skills.

A. Reliability: Definition

• The degree of consistency between two measures of the same thing. (Mehrens and Lehman, 1987).

• The measure of how stable, dependable, trustworthy, and consistent a test is in measuring the same thing each time (Worthen et al., 1993)

For example, if we wish to measure a person’s weight, we would hope that the scale would register the same measure each time the person stepped on the scale.

…

B. Validity

1. Definition:

• Truthfulness: Does the test measure what it purports to measure? the extent to which certain inferences can be made from test scores or other measurement. (Mehrens and Lehman, 1987)

• The degree to which they accomplish the purpose for which they are being used. (Worthen et al., 1993)

For a test to be valid, or truthful, it must first be reliable. If we cannot even get a bathroom scale to give us a consistent weight measure, we certainly cannot expect it to be accurate. Note, however, that a measure might be consistent (reliable) but not accurate (valid). A scale may record weights as two pounds too heavy each time. In other words, reliability is a necessary but insufficient condition for validity. (Neither validity nor reliability is an either/or dichotomy; there are degrees of each.)

http://course1.winona.edu/lgray/el626/MandEtext3.html

Most good certification exams begin with a blueprint.  The blueprint defines the topics and sub-topics that are likely to be included on the exam.  In some cases, a blueprint will be developed by a group of subject matter experts who are familiar with the knowledge and skills needed to perform the job.  In some cases these are the experts who will also develop the test items.  In other cases, a different group of subject matter experts will develop the questions.  Blueprint development can also be based on the learning objectives from course offerings that cover exam topics.  Once a blueprint is developed, it should be put out for review by subject matter experts.  Most often these experts are performing the job roles and they are knowledgeable about what is needed to successfully carry out the needed activities.

The blueprint areas are usually weighted for importance and an associated number of items to be written are assigned to the topics to be covered.  Publishing the blueprint and associated sample test items is usually done so that the candidates can prepare for the exam.  Sample items for well-constructed exams are of the same difficulty and quality level as items the exam candidate might expect to find on the exam.

While processes vary and item development might be in workshop format, or submission by subject matter experts or another process, the end result is a certification exam that will allow individuals who pass the exam to present themselves as having the expertise that the exam certified.

Certification exams are developed and taken so that there is a standard.  People who have passed the exam can say they have met this standard.  It allows for selection of providers who have a credential that informs the “buyer” that they have engaged a professional for a role in which they have shown their expertise by passing an exam.  An exam that has been put in place to test many people who want to establish they have expertise in a specific area.

Certifications are valuable and worthwhile.  But a certification is only as good as the exam that produced it.  Avoid these common pit falls and stay focused on the goal: an exam that tests the skills, behavior and knowledge needed to carry out the tasks of the role.  In the next installment, I will discuss recommendations for updating and revising certification exams.