Evaluating Multimedia -- Occupational Health & Safety

Evaluating Multimedia

Multimedia training is often slighted for not being as fun as games. But the outcome of multimedia training is to take, not play, a program that enables users to do something new or do something correctly.

By Dr. Terrell Perry
Sep 01, 2003

SO, you've made the decision to use multimedia as part of your overall training strategy. You can take the low-cost route and purchase generic off-the-shelf products, or you can expend more of your training budget on your own development. But do you know how to tell the difference between high-quality multimedia and multimedia that is substandard? (It may do the job, but not do it as well.)

What Does Research Tell Us?
Studies have shown that students using multimedia have higher achievement/performance ratings, higher retention, are given more consistent content, and they do it all in less time. The reasons given for these improvements include:

Multi-sensory input.
Self-paced learner control over the educational process.
One-on-one individualized training support.
Immediate interaction and feedback.
Constant, highly effective reinforcement of concepts and content.
Better researched, organized, and more succinctly presented content.
Greater attention to accuracy, detail, and completeness of content.
Learner mastery of content before continuing to the next unit/level.
Access to, and repeated interaction with, material.
More opportunity to apply what has been learned.

What Do the Professionals Think?
Professionals in the field of training look at a number of different characteristics to determine whether a multimedia training product will be effective or not. Some of the criteria for evaluating multimedia or Internet-based training are shown in Table 1.

Table 1. Criteria Used by Professionals for Evaluating Multimedia

1. Is the course easy to install, set up, or access on an Internet or intranet server using standard Internet or network protocols? Does it run properly?

2. Does the program include a sufficient amount and quality of information?

3. Does the program educate, inform, and provide insight to the viewer?

4. Is the content technically and factually accurate for the subject, discipline, or industry to which it is related?

5. Are scenarios, simulations, or exercises realistic to the actual job tasks or conditions?

6. Does the program enhance the instruction/delivery of the subject matter?

7. Do the direction, sound, editing, and photography work together and work consistently with the information to engage the viewer completely in the program?

8. Does the course present students with an overview that describes the purpose of the course and who will benefit from taking the course? Does it provide tips on how to use the courseware successfully?

9. Does the course address an overview, outline, and learning objectives for its various segments?

10. Is the course designed in a way that ensures users will learn?

11. Does the course present core competencies, performance standards, learning objectives, learning activities, and assessment activities to students in an easy-to-use, intuitive interface?

12. Does the course present syllabus information such as grading/certification policy, expectations for participation, due dates for assessments, and protocol for interaction between instructor/facilitator and students (and for students and other students)?

13. Does the course provide for instructor/facilitator/mentor feedback to student (and student-to-student feedback)?

14. Does the program capture the viewer's interest and spark discussion?

15. Does the program engage the student through novelty, humor, gaming,

testing/quizzing, discovery-adventure, and unique or surprise elements?

16. Does the program effectively and appropriately use media such as video, animation, music, narration, sound effects, and special visual effects?

17. Is the program designed for the intended audience?

18. Does it avoid being condescending, trite, pedantic, or too "cute"?

19. Are the reading and educational levels appropriate to the audience?

20. Do course materials communicate directly and consistently with the student, using the second person?

21. Is the program attractive and appealing to the eye and ear?

22. Are there large variations in volume between sound elements?

23. Are there large variations in quality among the various graphics, animations, or motion video elements?

24. Does the user have ample opportunity to engage program elements through his or her own input?

25. Does the program encourage just-in-time learning, question-answer, and problem-solution problem solving?

26. Does the course include and support learning activities that teach students, not just test them ("discovery learning")?

27. Does the course include a significant amount of practice activities for students?

28. Does the course include interactive learning aids, such as glossary and performance checklist tools?

29. Can users determine their own path through the program?

30. Does the course present clear, user-friendly instructions to the student regarding navigation in the multimedia learning environment?

31. Is there appropriate use of icons and/or clear labels so users don't have to read excessively or take tutorials to determine program options?

32. Is some type of evaluation used?

33. Does the course include performance assessment tasks with a supporting scoring guide?

34. Is mastery of each section's content advised before student proceeds to the next sections?

35. Is demonstrated performance required?

36. Does the student get a different assessment experience if the activities are repeated? (That is, are assessments programmed to randomize, or vary based on performance?)

37. Are student performance data recorded, such as time to complete, question analysis, and final scores?

38. Is the data forwarded to the course manager automatically?

39. Are report printouts available?

40. Does the course provide for innovative approaches to teaching and learning?

A Word about Games
Many people make the mistake of evaluating multimedia training as they would a multimedia game. To compare multimedia training to a game is like comparing apples to oranges: They're both fruit (I mean, multimedia), but that's where the comparison ends. Here are some of the criteria used for evaluating games:

Can the user easily understand the goal of the game?
Do you compete for points, against time, to uncover facts, or for something else?
Are the images used (graphics, motion video, animations) realistic and true to life?
If the game is based on a real life activity, is it realistic and accurate to the activity?
Is the game's user interface intuitive and easy to use without an instruction manual?
Does it use facts, statistics, reference material, or tools that are used in the actual activity?
Does functionality (the way the game works) change relative to adjustments made by the user? (For example, adjustments made to game settings from statistics, reference material, or tools.)
Is the basic plot or premise of the game sophisticated enough to maintain user interest, but not too complicated to play and enjoy?
Can progress in the game be saved and continued at a later date? Can it be played over time?
Is it a game played by an individual, against/with other players, or both?

Multimedia training is often slighted for not being as exciting and as fun to play as games. But the outcome of multimedia training is to take, not play, a program that will enable the user to do something new or do something correctly. Games, on the other hand, have only to entertain, with no requirement to train or teach. Some of the other notable differences are shown in Table 2.

Table 2. Multimedia Training-Game Comparison
Category	Multimedia Training	Multimedia Game
Purpose	To teach/train/educate to specific learning objectives	To entertain with or without a goal
Basis	Usually based on real-life tasks or conditions	May be based on real life or fantasy
Desired Outcome	Tries to change behavior; show learning gain	Tries to entertain and let the user have fun
Orientation	Logically outlines and arranges content; uses interconnected linear segments	Often uses free-play; usually non-linear in structure
Image Quality	Places less value in realism of graphics, animations, video	Places extremely high value in realism of graphics, animations, video
User Interface	Takes less liberty with program icons and navigation	Takes much more liberty with program icons and navigation
Use of Competition	Less use of negative reinforcement and punishment; greater use of feedback to shape desired behavior	Greater use of negative reinforcement and punishment; greater use of win-or-lose model
Reliance on Accuracy	Has a greater requirement for content to be technically accurate	Less need or requirement for technical accuracy of content; greater need for an accurate "look and feel" of play

What Should You Look For?
Evaluators should keep two important considerations in mind when evaluating a multimedia training product. First, even though all of us would like to make the learning experience as entertaining, fun, and painless as possible, our primary goal is to teach individuals to perform a job, task, or activities properly and safely. Second, all multimedia training is not created equal. Some products are more effective than others.

The checklist in Table 3 provides questions to ask when evaluating a multimedia training product. The more questions that you can answer yes to, the higher is the quality of the product.

Table 3. Multimedia Training Evaluation Checklist
Category	Criteria
Installation	Is the course easy to install and set up?
Content	Does the program include a sufficient amount and quality of information to educate, inform, and provide insight to the learner? Does it enhance the instruction/delivery of the subject matter?
Technical Accuracy	Is the content technically and factually accurate for the subject or discipline to which it is related?
Technical Quality	Do the direction, sound, editing, and photography work together and work consistently with the information?
Audience Identification	Does the program describe who will benefit from taking the course and provide tips on how to successfully use the courseware?
Performance Objectives	Does the course present an overview that describes the purpose of the course, the course outline, and learning objectives?
Instructional Strategies	Are learning activities easy to understand, and do they teach what the learning objectives say they will teach?
	Are course questions, quizzes, and tests easy to understand, and do they test what the learning objectives say they will tests?
Support Aids	Does the course present summary information, such as grading/certification policy, expectations for participants, due dates for assessments, and protocol for interaction between instructor/facilitator/mentor and students, and between students and other students?
Motivation	Does the program capture interest and engage the student through novelty, humor, gaming, testing/quizzing, and discovery-adventure elements?
Use of Media	Are the media selected for the program--video, animation, music, narration, sound effects, and special visual effects--effectively and appropriately used?
Tone	Do course materials communicate directly and consistently with the student? Does it avoid being condescending, trite, bookish, and too cutesy? Is the program properly designed for the intended audience? Are the reading grade and educational level appropriate for the audience?
Aesthetics	Is the program attractive and appealing to the eye and ear? (Or are there large variations among similar media types, i.e., volume differences between sound elements or quality differences among graphics, animations, or motion videos?)
Interactivity	Do users have ample opportunity to engage program elements through their own input? Does the program encourage just-in-time learning, question-answer problem solving, and learning activities that teach by questioning students, not just by questioning for evaluation? Does the course include a significant amount of practice activities for the student? Does the course include interactive learning aids such as glossary, word search, and performance checklist tools?
Navigation	Does the program present clear, user-friendly instructions to students regarding navigation in the program? Is there an appropriate use of icons and/or clear labels so learners don't have to read excessively or take tutorials to determine program options?
Assessment/Evaluation	Are section quiz, pre-test and final exam evaluations with appropriate feedback used? Is demonstrated performance required? Is mastery of each section's content advised before proceeding to the next section(s)? Does the student get a different assessment experience if the activities are repeated? (Does the user get different questions/exercises presented in different ways each time?)
Recordkeeping	Are student performance data recorded and printable?
Creativity	Is information delivered in a clear, interesting, and/or innovative way? Are innovative approaches to teaching and learning used? Does the course include a variety of learning media, such as text, graphic organizers, diagrams, graphics, charts, job aids, video, and audio clips?

Multimedia training should embody the principles of how adults learn. Although many of the principles and techniques from classroom instruction are appropriate for multimedia training, many are not, and these should be replaced by more effective methods that better fit the adult learner and the interactive computer environment.

This article originally appeared in the September 2003 issue of Occupational Health & Safety.

Product Showcase

SlateSafety BAND V2

SlateSafety's BAND V2 is the most rugged, easy-to-use connected safety wearable to help keep your workforce safe and help prevent heat stress. Worn on the upper arm, this smart PPE device works in tandem with the SlateSafety V2 system and the optional BEACON V2 environmental monitor. It includes comprehensive, enterprise-grade software that provides configurable alert thresholds, real-time alerts, data, and insights into your safety program's performance all while ensuring your data is secure and protected. Try it free for 30 days. 3