Evaluating the Effectiveness of Safety Training
Have the participants perform, and you will have a better understanding of the actual learning that took place.
AS safety professionals, all of us have a basic understanding of the effectiveness of our safety program--but what about our safety training specifically? Many people, unfortunately, see safety training as a compliance issue: something that has to get done, not necessarily something that is really going to change or impact safety performance.
Training employees about safe forklift operation and actual safe forklift operation may be miles apart. Evaluating the effectiveness of our training not only will allow us to see areas where the training needs to be improved, but may provide insight on ways to improve it, as well.
Using Reaction Surveys Wisely
Most of us are familiar with perception or reaction surveys typically given at the end of a class. These are also known as level-one evaluations by most trainers. They evaluate the training based on the participants' perception. Often administered at the end of a program, they typically ask questions such as: Was the training effective? Was the instructor knowledgeable? Were the handouts appropriate? Was the room comfortable?
While perception surveys can provide good information, you want to be sure the questions are relevant. Think about it--if the evaluation is given at the end of the class, will knowing temperature comfort levels benefit you after the class is finished? This should be determined before the class begins, or at least toward the beginning. Many times, there is nothing the instructor can do to change or adjust certain things. Unless the information is going to be used immediately or for the next class, why ask?
While asking whether the instructor appeared prepared or had good delivery may be appropriate, whether the instructor was knowledgeable really is not something that should be asked of the participants. Another content expert should determine the knowledge level of the instructor, not the students. If the students were in the position to be able to judge, then they probably don't need to be in the class.
Another common mistake when thinking about evaluation is not considering whether the class is going to be repeated. In other words, certain aspects in a one-time class may not be worth evaluating. Remember, one of the main reasons for evaluation is to be able to make adjustments or improvements on the next class. If there isn't a next class, you may want to reconsider the need to evaluate, or at least the need to perform a level-one reaction survey. Perhaps a higher-level evaluation (level three or four) would be more appropriate.
Higher-Level Evaluations
A level two evaluation tries to determine whether the participants learned anything. This is frequently done in the form of a quiz at the end of a class, often just before the level one perception survey. You may hear the instructor say, "After you complete the quiz (level 2), please complete the evaluation (level 1)."
Level two evaluation is the most common evaluation tool used. Most trainers see it as the way to demonstrate the training was effective. Just ask, "How do you know the training was effective?" and you probably will be handed the stack of quizzes. This level of evaluation can do a pretty good job for certain things. It can show that knowledge was gained and retained--at least, until the end of the class. The problem is that knowledge does not necessarily mean "learning."
If the goal of the training is the retention of information, for instance the flashpoint of a certain chemical, then a quiz may be the best evaluation. If a skill or task is being taught, however, such as the use of a gas detection meter, then a more effective measurement of learning would be to set up scenarios, drills, demonstrations, role-playing, or some similar evaluation that would demonstrate learning took place. If you can create a situation that is as similar to the expected real-world performance as possible, you increase the likelihood of getting a more effective evaluation. Think about the performance requirement in a cardiopulmonary resuscitation class. A student may be able to remember and select the correct answer on the multiple-choice quiz for the compressions-to-breaths ratio or the sequence of events when activating the EMS, but if you have her actually perform CPR on the manikin, you get a better indication of her skill level. The same is true of forklift training, lockout/tagout, or any number of training sessions: Have the participants perform, and you will have a better understanding of the actual learning that took place.
You will want to use a form that allows you to document performance so you can keep records for your training. Documenting the higher-level evaluations is unfortunately what sometimes makes a quiz so attractive. A quiz may be quick and easy, but you will gain a great deal more by documenting performance.
Training Transfer: Where the Rubber Meets the Road
Level three evaluation looks at how well the training actually transfers to the workplace or was there a change in behavior. This is fittingly called training transfer and is where the rubber meets the road.
If your job is just to provide training, you may have a hard time determining whether the transfer is taking place. If you have some control over workplace safety as well as training, however, it may be easier. Often, the manager, supervisor, or co-workers are the ones who have greatest control over training transfer. A 1992 book by Broad and Newstrom is a good guide for developing transfer strategies. (See the references are the end of this article.)
If you choose to evaluate the training at a level three, it will require a bit more planning but will be worth the effort. Think about the skills that were taught in the training, and you'll realize what this means.
Getting the supervisor involved in the evaluation is not only a good way to increase transfer, but supervisors may be best suited to actually do this level of evaluation. With forklifts, for example, having a supervisor perform the evaluation in the workplace often makes the most sense. They are available and probably have the skill to evaluate performance accurately. You must make sure the evaluator has the proper qualifications to evaluate the effectiveness of the training. If the evaluator is not operating the forklift properly and safely, he is probably not the right person to do the evaluation. You can see where getting the supervisor involved in evaluation may be a way to get more people on board.
Some skills that are taught in training will be used immediately, which may make level three evaluation easy. Others may not be used for a long time or may never be used--CPR or first aid, for instance. That may be more difficult to evaluate, and you may have to plan a drill in order to determine the effectiveness of the training. Another example of level three evaluation would be the annual lockout/tagout review that is required by the OSHA 1910.147 standard. The intent of this is to determine whether students learned and can perform the task, even after they are back at work.
Level four evaluation looks at the impact training had on actual results. Not can they do it, but are they doing it and did it make a difference. Has the training changed the way work is performed with improved results? This is where training is supposed to have its impact and this is where most safety programs want to be. For many this may seem like a daunting evaluation task, but surprisingly it is often the easiest. This is because the performance issue that may have prompted the training can be measured again after the training to determine the effectiveness.
If forklift operators have been driving too fast in certain areas or not using their horns appropriately, then all we need to do is see whether the performance has changed after the training and, most importantly, whether the new behaviors continue down the road. To be sure that it is the training that has made a difference, other variables have to be accounted for. For example, if in addition to the training management sends out a memo stating that any employee caught speeding will be terminated, the behavior change may be a result of that management memo and not the training. However, if the only intervention was the training, then it is easier to see that training probably caused the behavior change.
Calculating Return on Investment
Finally, there is some belief that a level five evaluation is the return on investment of training. To show ROI you must calculate the cost of the training, including your time and employees' time, as well as any other cost involved. This is compared to the savings from the injuries or accidents avoided.
While many professionals have a difficult time showing the return on investment for training, the safety professional has an advantage in that there are numerous statistics and data available to show the cost of workplace accidents, both in people and property. The OSHA Safety Pays Program (downloadable from OSHA's Web site at www.osha.gov) provides the cost of injuries, which in turn can provide valuable data for developing a return on investment evaluation. The cost versus return is set up as a ratio and may look something like for every dollar spent, 20 dollars were saved. While ROI may be more complicated than most people care to develop, it can be very useful when justifying a budget or hiring more staff.
While all trainers have to make decisions about the evaluations they will use, being familiar with the types of training evaluation available can be helpful. Consider evaluating your training at a higher level, and you will go a long way in showing the value of your training program.
Then, the next time someone asks you to demonstrate the effectiveness of your training, you can provide more than a stack of perception surveys stating the room was comfortable.
References
1. Broad, M.L. & Newstrom, J.W. Transfer of Training. Reading, Mass: Perseus Books, 1992.
2. Kirkpatrick, D.L. (1996). Evaluation. In Craig, R.L. (Ed.). The ASTD training and development handbook. New York: McGraw Hill.
3. Phillips, J.J. (1997). Handbook of training evaluation and measurement methods. Houston, Texas. Gulf Publishing Co.
4. Phillips, J.J. (1996). Measuring the results of training. In Craig, R.L. (Ed.). The ASTD training and development handbook. New York: McGraw Hill.
This article originally appeared in the June 2003 issue of Occupational Health & Safety.