Earlier this year, I spent some time researching the question of hard evidence for the effectiveness of simulation-based training. I was expecting to find such evidence, but in doing my research, what surprised me was the breadth of research across a variety of industries, all pointing to the same conclusion: enhancing traditional training with the use of simulations resulted in improved performance. I documented my research in an article, "Simulation-Based Training: The Evidence Is In", which has just been published on the Website of Chief Learning Officer magazine. Here's the original, unabridged version of the article:
Simulation-Based Training: The Evidence Is In
Pouring money into a theory is risky, especially when there is no documented evidence at hand to support that theory. An associate of mine who works for the Department of Defense was in search of evidence that proved simulations, games, and related training technologies improved performance in live situations. He was suggesting to his superiors -- both civilians and military officers -- that they spend precious research and training dollars on what appeared to them to be unproven ideas. The fact that these "unproven ideas" seemed, to my acquaintance, to quite obviously work, simply wasn't good enough.
As the co-founder of a firm that focuses on simulation-based training and serious games, I felt sure that hard facts existed regarding the effectiveness of using computer-based simulations for training. I began to survey literature in preparing to write this article, and what surprised me the most was not that I found exactly the kinds of research for which I was looking, but rather the breadth and sheer volume of such research. In every industry I examined, I found that people had already studied the use of simulations for training. In the reports I read, the results ranged from guardedly optimistic to wildly enthusiastic.
Having conducted the literature survey for this article, my belief is that, over the last few years, as simulation-based training has made the transition from high-cost, dedicated hardware systems to inexpensive, PC-based, virtual training solutions, and as a whole new generation of developers and managers has entered the industry, often with backgrounds in video game development, we have been unaware of the considerable research already in existence, documenting the effectiveness of what we were building.
This is not a comprehensive survey of the literature on the effectiveness of simulation-based training -- such a survey of only one industry, such as medicine, would have to cover dozens upon dozens of citations. This article is instead a quick overview of a dozen or so citations that demonstrate the breadth of research in this area as well as its surprisingly long history. To read more about this topic, use the references section for articles cited at the end of the article.
Pilot and Aircrew Operations
The Naval Training Systems Center conducted a meta-analysis (Hays, et. al. 1992) of 247 published articles, research reports, and technical reports studying the effectiveness of simulation for pilot training. Of these, 26 experiments were identified as having sufficient information for statistical meta-analysis, and 19 of these experiments focused on jet pilot training. Among jet pilot training studies, more than 90 percent of the experimental comparisons favored simulator and aircraft training over aircraft training alone.
In a study (Nullmeyer, et. al. 2006) of the effectiveness of virtual training for crew resource management (CRM) conducted with the U. S. Air Force 314th Airlift Wing, a low-cost, PC-based simulator using Microsoft Flight Simulator was designed to elicit the communication and crew coordination behaviors associated with instrument and visual airdrop missions. Treatment group students received a four-hour training profile before their first airdrop flight while control group students did not. CRM performance ratings during the first subsequent airdrop flight were significantly higher for treatment group students than their for control group peers. Higher performance grades in training records were also observed for treatment group students in all CRM skill areas through subsequent flights, with fewer sorties to criterion for communication, crew coordination, task management, and decision making for both navigators and co-pilots. For the two targeted skill areas of communication and coordination, simulator-trained student navigators required, on average, 10.2 sorties to achieve proficiency, compared to 11.8 sorties for their control group counterparts -- a reduction of 13.6 percent.
In 1969, a study (Abrahamson, et. al. 1969) conducted at the University of Southern California School of Medicine evaluated the use of a physical patient simulator, Sim One, in training anesthesiology residents. The study demonstrated that simulator-trained anesthesiology residents required a mean number of 9.6 endotracheal intubations to achieve a skill level high enough to perform four consecutive professionally acceptable intubations, compared to 18.6 intubations for residents not trained on the simulator to reach the same standard. The same study showed that simulator-trained residents achieved the most exacting evaluation criterion applied in 55 days of training, compared to 77 days for their non-simulator-trained counterparts -- a savings of 22 days, or over 28 percent.
More recently, a randomized, double-blind study at the Yale University School of Medicine's Department of Surgery evaluated (Seymour, et. al. 2002) residents trained in laparoscopy using a virtual reality (VR) system, Minimally Invasive Surgical Trainer-Virtual Reality, from Swedish firm Mentice AB. The study found that VR-trained residents performed significantly better during cholecystectomy (gall bladder removal) surgery than a control group not trained using the VR system. VR-trained residents worked 29 percent faster than control group participants and errors were six times less likely to occur during their surgeries. Control group participants were five times more likely to injure the gall bladder or burn non-target tissue and nine times more likely to fail to make progress for a minute or longer at some point during surgery.
A researcher at the Institute for Defense Analyses extracted 11 studies in which simulated equipment was used to train maintenance technicians (Fletcher 1998). These studies compared instruction with the simulators to instruction with actual equipment, held overall training time roughly equal, and assessed final performance using actual (not simulated) equipment. Over the 11 studies, the use of simulation suggested an improvement from 50th percentile to about 66th percentile achievement among students using simulation. The initial investment for simulated equipment averaged 43 percent of that for actual equipment, while the operating costs for simulated equipment averaged 16 percent of that for actual equipment.
At the request of the Canadian Air Force, the Canadian Forces School of Aerospace Technology and Engineering conducted a study (NGRAIN 2006) in which it adopted a virtual training system for what had been a two and a half day training course instructing students in the maintenance and removal of propellers for C-130 aircraft. The virtual training system consisted of CAE Simfinity's Virtual Maintenance trainer and NGRAIN's Virtual Task Trainer. After one day of study, all students passed the practical examination with an average score of 94 percent. This represented a 60 percent reduction in training time. Commenting on the study, a Canadian Air Force officer said, "Trainees can acquire knowledge faster when 3D equipment simulations supplement traditional methods."
The U. S. Army Research Institute for the Behavioral and Social Sciences and the University of Louisville examined (Shlechter, et. al. 1995) the instructional effectiveness of the U. S. Army National Guard's (ARNG) Reserve Component Virtual Training Program (RCVTP) -- structured exercises conducted in a simulation training environment, the SIMulation NETworking (SIMNET) training system -- to provide ARNG armor units with intensive training experience during their weekend drills or annual training periods. After 12 hours of training using RCVTP, units took 46.1 percent less time to complete their tasks while making 76.1 percent fewer errors and requiring 84.4 percent less coaching.
In 1991, the U. S. Army Research Institute compared (Bessemer 1991) results from 714 platoons that received conventional training in the Armor Officer Basic Course with 39 platoons that received training based on networked simulation using SIMNET and found that networked simulation both improved field performance ratings by 25 percent and saved 20 percent of time in the course.
Researchers at Rice University conducted a study (Lane and Tang 2000) in which the effectiveness of simulations for teaching statistical concepts was compared to the effectiveness of a textbook. Subjects trained with the simulation, developed at Rice, outperformed subjects trained with the textbook on seven out of eight critical questions, and were more able to recognize the key elements of ill-defined problems embedded in various real-world situations and apply the relevant statistical principles. According to the authors, "this provides support for the thesis that simulation is effective for training on educational and cognitive tasks (as opposed to tasks such as flying an airplane where simulation has been shown unequivocally to be effective)."
The Justice and Safety Center at Eastern Kentucky University studied (Eastern Kentucky 2003) the use of a mobile, trailer-housed firearms and judgment simulation system, Professional Range Instruction Simulator (PRISim) from Advanced Interactive Systems, for training law enforcement officers. After two hours of training using the system, officers placed an average of 31.6 percent more of their shots on target, and were 52.9 percent less likely to fire their weapons without justification. In the words of the authors, the system "appears to be beneficial in building and/or enhancing skills that are arguably the most important for the safety of the officer and others, i.e., accuracy, effective use of cover, avoiding the unintentional shooting or endangering of innocents and ensuring the shooting is justified."
Driving and Trucking
Long-haul trucking firms have been adopting simulators for training and are quantifying the results, according to an article in Heavy Duty Trucking magazine (Lockridge 2006). One firm, Schneider National, saw a 21 percent reduction in preventable accidents in drivers' first 90 days after simulation-based training, and its vice president of driver training and safety remarked that "one hour of training in the simulator is equivalent to three or four hours of training over the road". Another trucking firm, Bison Transport, saw its accumulated safe driving miles increase by 50 percent after adopting simulators for training, and its CEO noted that his firm had observed "an 83 percent improvement in mean time between incidents after simulator training for preventable accidents". Both firms use the Mark III and TranSim VS simulators from MPRI.
The University of Utah conducted a study (Strayer, et. al. 2004) of a pilot training program at the Utah Department of Transportation developed for snowplow operators, using the Mark II and TranSim VS driving simulators. The authors stated that, in the six months following training, "the odds of getting in an accident were lower for the group of drivers who received training compared with a matched control group who did not receive it". They also noted a 6.2 percent improvement in fuel efficiency for drivers receiving the simulation-based training.
In every survey and every industry examined, simulation-based training was seen to have a positive effect. In every case in which simulation-based training was directly compared to traditional methods, simulations were observed to be superior on some or all criteria. In every case where the costs of simulation-based and traditional training were compared, simulations were found to be less expensive, whether due to lower acquisition costs, lower operating costs, or lower costs resulting from more effective or faster training. In every case in which the author(s) of a study made a recommendation about the use of simulation-based training, they recommended its ongoing or expanded use.
In short, simulation training has demonstrably reached the point where questions of its fundamental effectiveness should no longer play a part in evaluating its potential use for any given project. In conducting such evaluations, the basic usefulness of simulation training can now be taken as a given, allowing project planners and decision-makers to focus their attention on their specific applications and how best to utilize simulation training in the most useful and cost-effective manner.
Abrahamson, S., J. S. Denson, and R. M. Wolf. 1969. Effectiveness of a simulator in training anesthesiology residents. Journal of Medical Education 44: 515-519. Reprinted in Quality & Safety in Health Care 2004 13:395-397. http://qshc.bmj.com/cgi/reprint/13/5/397.pdf (accessed January 10, 2007).
Bessemer, David W. 1991. Transfer of SIMNET training in the Armor Officer Basic Course (ARI Technical Report 920), Alexandria, VA: Army Research Institute.
Eastern Kentucky University. College of Justice and Safety. Justice and Safety Center. 2003. The evaluation of a mobile simulation training technology—PRISim. http://www.jsc.eku.edu/reports/PRISimReportFINAL.pdf (accessed January 10, 2007).
Fletcher, J.D. 1997. What have we learned about computer based instruction in military training? In R.J. Seidel and P.R. Chatelier (eds.), Virtual Reality, Training's Future? New York, NY: Plenum Publishing.
Hays, Robert T., John W. Jacobs, Carolyn Prince, and Eduardo Salas. 1992. Flight simulator training effectiveness: a meta-analysis. Military Psychology 4(2): 63-74. http://www.leaonline.com/doi/pdf/10.1207/s15327876mp0402_1 (accessed January 10, 2007).
Lane, David M. and Zhihua Tang. 2000. Effectiveness of simulation training on transfer of statistical concepts. Journal of Educational Computing Research 22(4): 383-396. http://baywood.metapress.com/link.asp?id=w9gw5m9cuqvt1e0r (accessed January 10, 2007).
Lockridge, Deborah. 2006. Simulated training. Heavy Duty Trucking, October. http://www.heavydutytrucking.com/2006/09/044a0609.asp (accessed November 29, 2006).
NGRAIN (Canada) Corporation. 2006. NGRAIN and CAE help Canadian Forces increase training throughput. http://www.ngrain.com/solutions/casestudies/articles/NGRAIN-CDN_Forces_Train_60_percent_Faster.pdf (accessed January 9, 2007).
Nullmeyer, Robert T., V. Alan Spiker, Katharine C. Golas, Ryan C. Logan, and Larry Clemons. 2006. The effectiveness of a PC-based C-130 crew resource management aircrew training device. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2006 Conference Proceedings. http://www.iitsec.org/documents/Trng_2807.pdf (accessed January 9, 2007).
Seymour, Neal E., Anthony G. Gallagher, Sanziana A. Roman, et al. 2002. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Annals of Surgery, October 236(4): 458–464. http://www.pubmedcentral.nih.gov/picrender.fcgi?artid=1422600&blobtype=pdf (accessed January 11, 2007).
Shlechter, Theodore M., David W. Bessemer, Paul Nesselroade, Jr., and James Anthony. 1995. An initial evaluation of a simulation-based training program for Army National Guard units. Research Report 1679. U. S. Army Research Institute for the Behavioral and Social Sciences. http://handle.dtic.mil/100.2/ADA297271 (accessed January 10, 2007).
Strayer, David L., Frank A. Drews, and Stan Burns. 2004. The development and evaluation of a high-fidelity simulator training program for snowplow operators. Proceedings of the Third International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design: 464-470. http://ppc.uiowa.edu/driving-assessment/2005/final/papers/68_DavidStrayerformat.pdf (accessed January 10, 2007).
Turpin, Darrell and Reginald Welles. 2006. Analysis of simulator-based training effectiveness through driver performance measurement. Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2006 Conference Proceedings. http://ntsa.metapress.com/link.asp?id=ebe0e1hlf6adc67q (accessed January 10, 2007).