Archive for November, 2018

Innovations in Learning 2018

November 20, 2018

Of late there seems to be renewed interest in finding new ways to make learning happen. It isn’t just new ways to deliver training, which of course is good, but what I’m coming across is more innovative ways of helping learners absorb and apply content. This is exciting because there are many areas to explore as learning professionals seek to find new and better ways to make learning, and more importantly, the application of learning happen.

Now more then ever learning professionals are working toward active learning that involves engagement of both intellectual and physical activities. And with all our modern technology we can share what is being done, the results and learnings to the greater community of learning professionals. We may actually be at a point where we can better affect how learning can be applied for better results.

While research shows that interactive learning is superior to passively listening or reading, most corporate learning events continue to offer passive instruction. Research is also showing that not only is interactive learning superior to passive learning, it is also finding that small group interactive learning improves long term retention of concepts that can be applied to job responsibilities.

https://www.brookings.edu/blog/education-plus-development/2018/05/30/learning-about-learning-meaning-matters/

In the May 30, 2018 post “Learning about learning: Meaning matters”

Andres S. Bustamante and Kathy Hirsh-Pasek discuss the work of Professor Noah Finklestein of the Physics Department at the University of Colorado. He and his colleagues are working to transform the approach to educating undergraduates.

“…students are both engaged in hands-on minds-on learning and are much more likely to retain the content than their peers who were talked at or lectured to. Irrespective of age, people learn best when learning experiences are active, engaged, meaningful, and interactive.”

Most course developers in business environments are tasked with developing instructional content with templates that are in need of updating. At this point most corporate course developers are using new media and others forms of instructional tools and are moving away from lectures and slides. But the instructional templates that they are often required to use are in need of being refreshed and updated. The good news is that academic institutions are working to deliver content that will hopefully provide templates for the future development of business learning events.

“In a move away from the classic lecture format, Dr. Finklestein is using an ‘interactive-engagement’ approach where students work in small groups and apply the content in computer simulations and hands-on activities. The data show that student-learning outcomes of the most seasoned lecturers are equal to the lowest performing interactive-engagement classrooms, and when done well, the interactive-engagement format vastly outperforms the top lecturers with students learning more than double the physics content.”

At the beginning of November 2018, several institutions of learning jointly sponsored an event at which some of the new educational practices in use at those institutions were presented, discussed and practiced during the day-long session. The event contained three tracks, “Deepen” “Design” and “Dissent.” All the attendees rotated through each of the tracks during the day. The planners and organizers of the event included faculty, staff and students from the several institutions of learning that sponsored the event.

http://olin.edu/the-wire/2018/remaking-education-offers-new-perspectives-tools-reforming-education/

One notable difference from other types of events is that for this event students played a major role in designing and facilitating the sessions. Participants in the sessions were composed of people from various backgrounds all interested in advancing learning.

Attendees had an opportunity to participate in collaborative, hands-on activities, mentorship, contextual learning, and to hear about and participate in discussions of how these activities can impact educational practices going forward. Not only did the event talk to how to change how learning can be presented, the entire program modeled the educational practices that were being presented and advocated.

In the “Deepen” session participants had an opportunity to learn by doing and also to recount episodes from their own life where they gained knowledge from learning that they achieved through experience. During the “Design” session attendees experienced hands-on learning as they worked together in small teams to produce a product that required cooperation and collaboration. In the “Dissent” session participants were tasked with identifying ways that pieces of the current educational system could be removed and replaced with different and better methodologies to achieve the same goals. The idea was to better meet the needs of all learners, instructors and the institution to achieve the needed end results. The participants worked in small teams to develop their recommendations.

The event was all about presenting new ideas about how education and learning can occur. I hope that there will be follow-up information about how successful the people who attended the event were in implementing the concepts presented within their organizations. An understanding of what worked, what didn’t, and why will help to move new concepts and techniques forward.

Writing Complex Multiple Choice Questions

November 7, 2018

The last few years have seen a general improvement in the quality of learning events offered for business learners. We have learned that chunking content, offering hands-on activities, and self-tests improve retention. Even more importantly it improves how well learners apply that content. While all of this is great, there are still some improvements to be made. I would like to address how we develop test questions for large populations and provide some recommendations for developing more complex items.

I have often heard course developers complain that because of the need to develop multiple-choice questions they are unable to develop questions that actually test what they believe the learners need to be able to do. Hopefully, this blog and other posts will begin to provide more information about how they can more effectively develop multiple-choice questions.

Developers are required to use multiple-choice questions in most cases because we don’t have the bandwidth to read and review items for large groups of test takers. Given that requirement, there are ways to improve our multiple-choice items. Multiple-choice questions don’t necessarily mean simple questions. There are numerous ways to ask multiple-choice questions that allow for developing higher order items. 

Here I will go into a little bit of detail on a few methods that have a good success rate. As always, if you have any questions, please leave a comment below.

Pictures or graphics – If the test delivery tool allows for including graphics, they can be used in questions and/or response options. Show pictures of several items in each response choice, with only one answer choice showing all the correct matches.

Matching – ask a question that requires that the test taker match items in two columns. Include more items in the second column so that the match for the last item in column A doesn’t fall into place. Response options should list the letter (or number) of several items in each answer choice. Only one answer choice would list all the correct matches.

Example: Match the items in column A with the item in column B that is needed to operate that item. Note column B contains extra options.

Column A             Column B

 

A1. Clock with plug B1. Electricity
A2. Car B2. Gasoline
Etc. B3. Oven
  Etc.

 

Answer Options:

  1. A1 and B1, A2 and B2 etc.
  2. A1 and B2, A2 and B3 etc.
  3. XYZ and DEG, GXT and TOQ etc.

Putting a machine or other physical product together

If learners have been taught how to repair, fix or configure a machine a test question can ask in which order a machine should be put together. The answer options could show different parts that need to be installed. The question can ask the order in which to install the component parts.

Multiple Step Process – The first part of the question describes a situation, process or problem that requires several steps to resolve. The learner is asked for the steps and the order in which to complete them. A list of steps is then presented for the test taker to use to resolve the situation.

With this type of question it can be stated that there are many options that are not listed, explain that the test taker should only consider the options listed to respond to the question. Response options list the letter or number of several items in each answer choice; only one answer choice lists all the correct matches in the correct order.

Example: In which order should a person begin to plan for a long trip? There are many options that are not listed here, consider only the options listed to respond to this question.

  1. Determine the trip distance
  2. Decide on the mode of transportation
  3. Identify the costs of the different options available
  4. Identify the amount of time required for each option
  5. Find a friend to travel with to help defray the costs
  6. Consider all factors to find a good time to travel

Answer Options:

  1. 1,2,3,5
  2. 2,4,1,3
  3. 6,5,2,3
  4. 4,3,2,1

Provide a short description of a process – Describe a process and ask how it can be applied, or what can go wrong, or any other concept that the learner needs to know. Response options list the letter or number of several items in each answer choice. Only one answer choice lists all the correct matches to the content the learner needs to know.

Ordering – In this technique a question can list several steps in a process. The test taker is asked which steps need to be taken and in which order, to achieve a result. It is a good idea to list more steps than are needed. Mention in the body of the first part of the question that not all steps listed will be needed. Response options list the letter or number of several items in each answer choice; only one answer choice lists all the correct matches in the correct order.

Example: What steps need to be taken and in which order, to develop a KWPQ? NOTE: There are more steps listed than are need.

  1. do the thing
  2. list the thing
  3. put the thing in the place
  4. start the thing
  5. set the timer for the thing
  6. display the thing
  7. process the thing

Answer Options:

  1. 1,3,5,7
  2. 2,1,7,4
  3. 6,2,3,5
  4. 4,6,1,3

These are just a few ways and there’s a lot more to the craft of successfully developing tests with multiple-choice questions. But hopefully the above thoughts will help. As always, I’m glad to answer questions.