Developing Certification Exams

There are many factors to consider when implementing a certification exam program. The benefits of a well-developed and implemented program are many, ranging from well-prepared employees to increased profits due to a knowledgeable team. The tests themselves are most often a revenue generating process as many different populations can seek to be certified. But getting there requires multiple steps to ensure a quality exam.

It is critical to ensure that each step in the development process is well planned and executed. Below is an overview of steps that need to be taken to ensure a quality outcome. There are other steps that are followed by different development programs. However these are the major steps and processes.

• Begin by identifying the end results needed. What do the people who receive the certification need to be able to do.
• Once the end results have been identified, the steps needed to produce the end results have to be documented. For example, if the test covers driving different types of vehicles (perhaps cars, trucks, motorcycles, golf carts or others) include all the steps needed to drive each type of vehicle. This is known as a task analysis.
• Next a blueprint is developed. A team of experts most often develops the blueprint. The experts who develop the blueprint generally include practitioners and others who have expertise in the area. The blueprint describes each component of the task analysis. The blueprint identifies each area that will be tested by the exam. Most blueprints include the number of items for each objective that will be included on an exam. They also usually assign a weight to each topic area. Multiple examples of templates are available on line. Here is just one example.

https://www.nursingworld.org/~4acd24/globalassets/certification/certification-specialty-pages/resources/test-content-outlines/familynp-tco.pdf

• Once the blueprint is developed it needs to be validated by other subject matter experts (SMEs). Note that it is not expected that one person will have expertise in all areas covered by a full exam.
• At this point the item writing process begins. The very first step is to agree on the blueprint. Most often there are changes. Additions and some deletions can also happen, as topics may not be appropriate for the specific exam, may be covered on another exam, or might make the exam too long. Also there may be changes to the percentages and other edits before the team agrees on who will develop which items and the number of items each person will write.
• Handouts for the types of items to be included on the exam (for example multiple-choice, matching), how to write items, and what should and should not be included in an item are generally passed out and reviewed before participants begin writing the items. There are often handouts to explain how to input questions into the item development program used for item input.
• Here is an excellent set of guidelines for exam writing

http://www.cedma.org/blog/certification-exam-item-writing-best-practices

• It is important to let item writers develop just a few items (about three) and stop so that their items can be reviewed for feedback. Generally letting them write for no more than 30 minutes works well. It is also a good idea to review with the full team so that everyone hears the feedback. For a large group, the exam facilitator may want to just review one item written by each developer. Have them post their items in a way that allows for only the facilitator to see them. The exam facilitator should choose items to review so that different areas that need attention can be discussed.
• Once the first review is completed it is time to let the writers write more items. The exam facilitator needs to continue to review the developers’ items so that if they are making errors the facilitator can let them know what needs to be fixed. Usually a recommended rewrite helps. If the facilitator doesn’t know the subject area, they can use a gibberish word that will allow for the needed concept to be explained.
• After a period of time (usually two hours works) take a break and then begin to review the items. The facilitator can provide feedback on test development techniques and the developers need to provide feedback on the content of the questions. If there is agreement that the question is good, then it can be approved. If there isn’t agreement, the team can try to repair it. But in some cases it is better to delete the question, as time is often a factor in getting the number of questions that are needed and approved completed in the allotted timeframe.

Once the development workshop is completed the questions are sent out for review. Each question needs to be reviewed by multiple reviewers. Also each reviewer should only review a limited number of items to ensure the confidentiality of the exam.

That’s all for now. A future blog will cover additional information about certification exam development. Please let me know if you have comments or questions.

Surveys: A Short And Simple Approach, Part One

I believe in using surveys. Surveys can provide valuable information with very little effort. Surveys come in all sizes. But surveys that are too long or too intrusive often won’t return the wanted or needed data. Most organizations and individuals don’t use surveys often enough to get meaningful data for decision-making. A quick and simple instrument can dramatically help to provide actionable information for decision-making.

Why might you want to survey a population? More importantly, what do you want to learn? Knowing what you want to know before you survey will lead to more effective results and more useful information. If you wait to see what you get and then try to figure out what you learned, you might not learn anything.

Perhaps you want to offer a workshop on a particular topic, but you aren’t sure that enough people would be interested. A survey administered to members of your target population might well help you decide whether or not to develop that workshop. Or perhaps you want to offer a new product or service in your establishment. Again, a survey might give you the information you need before you invest time and money in something that might or might not be of interest to potential customers.

Like anything else, surveying has proven methodologies associated with it. It is a science like any other science. There are procedures that work, but many of the most commonly used methods provide little to no useful information.

Many people are of the opinion that surveys are used to gain information regarding complex or critical decisions. That is true, of course. But there are other ways to use surveys for decision-making that take a minimum amount of effort to deliver results that are well worth it.

The first question to ask when developing any survey is, “What is the desired end result?” The survey results are a step towards getting to that actual end result.

Once the desired end results have been defined, most people go on to defining the questions that need to be asked. However, I suggest that the next step should be answering the question “What is in it for the survey taker?” You can ask questions that your survey population will answer, but it is important to ask questions that your population will want to answer. If you haven’t done it already, here is where you want to define the members of your target population(s).

Once you have identified the needed end results, what is in it for the participants, and who the participants are, you can then construct your survey items.

Come back next week for some tips and strategies for building your survey questions!

Better Resume Development Through Teamwork

Here are some of my thoughts on ways to develop or update your resume. Working with a colleague who also needs to update their resume can allow you both to effectively update your resumes. It is important to have a current resume even if you are not currently looking for another position. There are many good reasons to have a current resume. Maybe your manager wants to present expertise in their group; maybe you will be invited to speak to a class about what you do. There are many other circumstances in which a current resume is a good thing to have, so take the time to update your resume. You won’t be sorry.

To begin with, keep in mind that your resume is what you are presenting to the reader about yourself. It is what you are saying about yourself and your background. Therefore it is unwise to say how great you are, as the reader won’t be impressed with you just saying you are the best thing since sliced bread. You need to share accomplishments and results that tell your readers what you have done rather than having a statement that says you are wonderful but doesn’t explain why.

Maybe you think you need help in presenting your accomplishments. One way to get this help is to work with a colleague to identify what you each have done that shows experience and success. Here are some things that I recommend you do. Starting with documenting everything as you go along. You may not use everything, but you won’t know what you want to keep until you are done.

In general I have found that people find it much easier to talk to someone else’s accomplishments instead of their own. Here are some pointers to consider:

• List 3 work related things that they did in their current role that resulted in successful outcomes
• Describe each of the outcomes; explain what made them successful
• Explain how the knowledge and skills used to create each outcome could be applied to one or more different situations. Describe some of the different situations
• For example your colleague created a new process for handling requests coming in to the organization. You can talk to how the process could be used for other types of requests or inquires.
• Now look at what your colleague documented about your successes. Is there anything you can add to what they said about your accomplishments? What did you do that was different but yet could also be applied to another situation?
• For example you documented a process and identified where things could go wrong and the steps needed to mitigate issues
• When you are both satisfied with the 3 activities from your current role go ahead and add as many additional key accomplishments for your colleague as you can. Use the same process.
• Review the accomplishments developed for you. Are there any additional accomplishments that you want to add that your colleague may not be able to document? If yes, go ahead and add them using the same process.
• Next move down through each of your prior roles. It is likely that you will have to develop the accomplishment statements for yourself as your current colleague most likely didn’t work with you in your prior roles. Use the same techniques you used for developing statements about your colleague. Maybe you only want to list a few accomplishments or maybe even one for a previous role. That is OK; go ahead and work through your prior roles.
• When you have completed this activity, which at this point is still a draft, review what you have. Are there things that you don’t want to do again? If yes, delete them. You do not have to list everything you have done in the past if it isn’t something you want to do again.
• Do you like the way these statements are presenting you? If not work with your partner to modify them until you are comfortable with the way the statement presents you.
• Step back and consider whether there are accomplishments that are not yet listed. Work with your partner to document them.
• When you are comfortable with your success statements it is time to consider if you have too many. A resume should be no more than two pages, with white space for notes. Consider that you also need to add your education. You may also want to add a summary statement at the top.
• Once you are done you may want to share it with others to get their feedback. Does it effectively communicate what you want a reader to take away from your resume? Essentially you are field-testing your resume.

I hope you find this process helpful and that you wind up with a current resume that you can add to as time goes on. As you add things, remember that a two-page resume is ideal. If need be, find line items to delete.

Good Luck!

Writing Certification Exam Questions

This blog talks about some of what goes into the development of a certification exam. It is not intended to cover every aspect of exam development. If you have specific questions or want more detail about certification exam development, let me know and I might cover your questions in a future blog. There is much more that can be said and I plan to present additional blogs on this topic.

If you have expertise in an area where a certification exam is planned, you may be asked to participate in the development of the blueprint or exam questions. Blueprints are developed to guide the development of the exam questions. In general, the people who participate in the blueprint development do not also participate in the development of the exam questions based on that blueprint. Participants in the item development process use the blueprint to guide the development of questions. In addition to item developers, people knowledgeable in an area are also invited to review questions. Reviewers are assigned a subset of questions and asked to provide feedback. Generally several people are asked to review each of the questions.

The blueprint developers determine how much of the test will be devoted to each topic and the question developers review and revise as necessary. Generally speaking the item developers review topic areas and validate the assigned percentages. But, these are guidelines. It may be decided during item development that more or fewer questions on a topic are needed.

While there are many ways to develop certification exams, the key goal is to be able to certify that someone who has completed the exams for a specific area can be certified by the exam organization to be proficient in that area.

In general, certification exam item development processes require sufficient questions for multiple versions of the exam and also a practice test. The practice test is generally made available for review prior to taking the actual exam.

Most people who are involved in the development of exam questions have expertise in the subject areas covered by the exam. Very few of them have expertise in learning or exam development. Good question development is based on science. The science of how people learn and the application of that learning is an important element of exam development. This leads to better outcomes for both the exam takers and the organizations that might employ them.

In developing test questions it is important to consider what people will need to be able to do on the job. Too often questions ask for information that is a small component of the larger needed area. For example a question might ask which tools a service tech should have in their toolbox. However, knowing which tools are needed doesn’t indicate whether or not the test taker knows how to use those tools. There would need to be additional questions about the use of the tools.

One of the best ways to test the application of knowledge is to present a scenario that describes a situation. Then ask a series of questions that test the concept or skill needed to perform the work. Often the test delivery tool is limited to multiple-choice questions. However, there are ways to test complex activities or concepts using multiple-choice questions. For example a question can be posed about the order in which steps need to be taken.

Example: A tech needs to determine why a product isn’t working properly. What steps does the tech need to take and in which order?

Steps: (Which 4 steps are needed and in which order)
1
2
3
4
5
6
Answer choices:
A. 1,2,3,4
B. 3,1,5,2
C. 4,5,6,1
D. 6,4,2,1

A small scenario can be described and several questions about the scenario can be developed.

For example: Jane and Joe are driving on a road in an unfamiliar area. They need to get to a destination by a certain time to meet with the manager to discuss the services their company offers. They encounter a detour that is going to add time to their trip. What steps do you recommend they take and why?

Steps: (Which 4 steps do you recommend)
1
2
3
4
5
6
Answer choices:
A. 1,2,3,4
B. 3,1,5,2
C. 4,5,6,1
D. 6,4,2,1
Why do you recommend these steps:
A.
B.
C.
D.

Regardless of the exam delivery tool, you can – and should – develop items that ask about end results. Some delivery methods provide more question format options than others, but by keeping the necessary end results in mind, you can develop questions that will test what is required on-the-job.

Less is More

As a course developer you want to cover each topic and overall module content sufficiently to provide learners with enough information for the job or real life situations. But, what should you include and what should you exclude? You want the learner to be able to perform the work, but you shouldn’t try to teach them everything there is to know about the topic. You want to ensure they learn what they need to know to do the work but not get lost in a forest of information. So how do you decide what to teach them and how much of each topic to cover?

Recommended steps:
1. Write objective statements
2. Develop questions which will test the learners’ ability to perform the objectives
3. List what learners will need to be able to do
4. List what they will need to know for successful performance of the objectives
5. Outline steps that need to be taken to do the work
6. Map each step to “need to know” statements
7. Identify any problems someone might encounter while completing each step
8. Add the steps that need to be taken to address identified problems to the outline of steps needed to do the work
9. Add examples that show how it is done

Add whatever words are needed to communicate these ideas but don’t add anything new at this stage. This is you minimalist unit of instruction.

Ray Jimenez has a wonderful short piece addressing these concepts.

http://archive.constantcontact.com/fs130/1011065179978/archive/1122903861032.html
Cut the Crap!!!
Ray Jimenez, PhD

As Ray Jimenez says “Simplifying your content is a conscious design choice. It means getting into the shoes of your learners and including only what you have thoroughly assessed and determined they really need. What’s not necessary is dropped and what is retained are only the stuff that matters.”

Now look at the questions you wrote for the objectives. Do they cover everything you have outlined? Is there anything that needs to be added to either the questions or the outline based on this review? If yes, add what is needed. What you have now produced is your draft lesson. If possible find one to three members of your target population to pilot the lesson. If you don’t have members of your target population available to field test the training, find someone who doesn’t know the topic and ask them to work through the instruction. Sit with the field test participants as they work. Ask them to identify anything that isn’t clear or if they have any questions. Consider what they say and make revisions that you believe are necessary. Unless you believe something really should be revised immediately, collect and compare all field test comments before making any revisions, and then only make revisions that are critical for understanding the objectives.

This may seem a bit involved but like anything else, a good foundation will prevent or decrease problems down the line. Later on in the process it will likely be more difficult to make revisions.

And following my own advice from this mini lesson, here is where I will stop and ask for feedback. Please provide feedback, questions and comments.

Thank you!

10 Steps to Design Effective Instructional Materials

This is (mostly) a check list of what I believe are the key points for instructional designers to remember. Use these before starting to develop a new instructional product. Similar to reading a reminder or an update about good nutrition, good sleeping habits or getting enough fresh air and so forth. This post addresses the steps that I think are important to winding up with quality instruction.

What’s old is new again or dressing up or modifying an old concept can lead to increased success. I am cheering in the stands because leaders in the field of learning have recognized and publicized the fact that (once again) less is more. Yes, some of us know this is a concept that has been around for a long while, but it is now getting prominence.

Give people small units of instruction with examples and an opportunity to practice components, provide feedback to the practice exercises, and then explain how to apply the learning to real world activities. Small units of instruction can be linked to each other to create a course. Okay, so here is a checklist for developing small (20 minutes or less) units of instruction.

1. Begin with the end in mind, document what you want the learner to be able to do
2. Develop test questions that will allow the learner to demonstrate what they are able to do
3. List the steps needed to accomplish the end results
4. Identify the order in which steps need to be completed. Showing which steps can be completed in any order and show which steps must be completed in a specific order
5. Explain the gotchas, list the things that could go wrong and explain how to avoid them, how to fix them, or how to mitigate them
6. Develop instruction for each step, explain why it needs to be done, why it needs to be done in a specific order (or doesn’t need to be done in a specific order)
7. Develop several examples for each step
8. Develop one or more questions that will allow the learner to test if they are able to perform the action needed
9. Find a member of the target population to go through the instructional unit while you take notes on what works and what doesn’t work
a. Note: If a member of the target population isn’t available have someone else work through the training and identify what might be confusing or missing
10. Modify the materials based on findings from the evaluation of the unit of instruction

This might look like a lot to do, but doing something right the first time will produce the product you need without having to revise or replace parts because the product doesn’t work. A learner should be able to complete each unit of instruction you develop in 20 minutes or less. The development process should also take less time once you gain proficiency in using this process.

I hope that you will use this checklist and see how it works for you. Please comment below!

Interviewing to Hire the Right Talent

While we are not fully sure where the economy is headed, all the signs say things will continue to get better and organizations are actively hiring. It appears that there is a dearth of qualified applicants in many fields and hiring managers are struggling to find the right people for their openings.

Hiring managers use many different strategies to select new hires. One of the most common strategies is to find applicants who present as having similar backgrounds and personalities to the interviewer, the manager, or to the culture and personality of the company and the team. Many managers believe that anyone can be taught the technical aspects of the job and therefore personality is the important factor to be considered in making hiring decisions. While making sure someone will blend into the organizational culture is certainly a key factor, it usually isn’t enough to ensure a good match for both the company and the individual looking for a position.

While many hiring managers or members of interview teams know how to conduct interviews that will result in a good match to their openings, some do not. Too often the interviewer doesn’t formally prepare for the interview. They read the resume and then will go into the interview session and wing it. Others will jot down some notes about aspects of the individual’s resume that they want to explore. This is a good start, but there is more preparation needed if the hiring team wants to improve the odds of a good hire.

I agree that it is important to hire people who will fit into the organization’s culture. I believe there are several factors that need to be considered when hiring new employees. An individual’s resume might list subject matter knowledge that is required for the position but it is also important to find out how the individual applies that knowledge. For example, if the position has strict standards that need to be followed and the interviewee has had good luck with improvising in similar situations, that person may not fit even though – on paper – it looks like they do.

It is my belief that hiring decisions need to take several factors into account. Also, each interviewer needs to prepare for the interview. People who will be interviewing candidates should all use the same template to note the responses to the prepared questions but also should have an area to make personal comments. Team members should share any issues they uncover with any of the interview questions as soon as possible so that all interviewers can make adjustments to the common questions. Each interviewer should prepare additional questions specific to their interests and should ask these questions of all the candidates that they interview. There are many good recommendations on-line for interview questions. Here is one example and many more are available:

https://www.thebalance.com/best-interview-questions-for-employers-to-ask-applicants-1918483

It is very important to take notes during the interview, don’t depend on your memory. Also, by taking notes during the interview it allows you to interview several applicants in succession. You can move onto the next applicant and not be concerned about remembering the specifics of each interview.

I strongly recommend against leading questions (e.g., I really like to do XYZ do you agree? Don’t you think that…) as this might indicate the desired answer to the interviewee. The interviewer should allow the applicant to speak without saying anything themselves. It is important to be attentive and to show attentiveness without encouraging or discouraging what the interviewee is saying.

Prior to conducting interviews create a list of specific competencies and knowledge areas that the individual hired will need to have. If several people will be conducting interviews it is a good idea to work as a team to create the list. However, it is best to get the hiring manager’s sign-off on the list of questions to be used.

I recommend the following information be gathered during each interview session:

• Interviewee name
• Area sought, for example: specific standards
• What applicant said to demonstrate evidence they have the requirement
• Comments

Also include summary thoughts

Adding new members to a team is very important to the success of an organization and it is important to take the time to prepare and to review results on your own and with all the people involved in the interviewing process and of course with the individual who will be making the hiring decision.

Developing Exam Questions in a Business Environment

In a business environment it is important for the management team to know how prepared and capable their teams are to perform their job roles. Whether teams interface with customers, clients or internal groups management wants to know that they are able to perform their job roles. Employees are often asked to take an exam or multiple exams to determine their proficiency in the areas where they need to show their capability.

In most cases organizations call upon employees who are knowledgeable in a subject area to develop exam questions. While subject matter expertise is required for developing good exams; development also requires knowledge of how to develop effective exam questions. The most common type of exam question developed is the multiple choice question. Usually a question with 4 or 5 answer options is developed.

There are several factors which need to be considered in developing exam questions. This paper will address just a few of the common considerations. Exams in a business environment are different than exams used in a K-12 or university setting. In a business environment people need to produce an end result. Exams need to test people’s ability to produce the end results required by their job role. The subject matter experts developing the exam questions need to be provided with guidelines, examples, tools and references to help them develop effective questions.

Guidelines

The guidelines should cover what should and what should not be done in developing questions. For example the exam sponsors may want to know if the exam taker can perform a complex process. It is important to have the exam mirror as closely as possible the environment that they will be working in on the job. Therefore if on the job people have references that they use the exam might provide access to the references or ask questions about which references are needed and how to access them to perform the activities.

Other points to cover in the guidelines include topics like the number of correct responses, whether two concepts can be examined in one question and explanations about why certain types of questions or responses should not be used. For example the use of humorous or give away incorrect responses. It is important to decide if it is important to examine the concepts by themselves to ensure what the exam taker knows. In some cases both areas need to be known together so it might be fine to examine them within the same question. In other cases the two concepts may stand alone and if an individual gets one right but not the other neither the individual nor management will know where they might have a gap in knowledge.

Also exam developers need to be told to be careful so that content in one question doesn’t provide the answer to another question. There are many other topics that need to be covered in the guidelines and therefore I recommend that exams be developed jointly by both a subject matter expert (SME) and an exam development consultant. An exam development consultant has expertise in how to develop exams and what should and should not be done in the development process.

Examples

Providing SMEs with examples of what good and poor questions look like along with why they fit into either category provides instruction on what to do and not do in developing questions.

Tools

A template that an exam developer can fill in makes it easier to develop questions according to the guideline and also provides a structure for development of the questions.

References

References such as course materials, manuals, or marketing materials need to be given to the exam taker and should be the source for the exam questions. The test taker can review what they got wrong and also read more on topics they need to know.

Feedback

It is important to provide information about why incorrect responses are wrong. This allows the exam taker to understand why a response is correct or incorrect.

While there is much more that can and will be said about exams this is all that I will cover in this post.