Effective Group Training With Computer-based Virtual Environments

 

The National Institute for Occupational Safety and Health (NIOSH) Pittsburgh Mining Research Division’s computer-based simulations in virtual environments are used in many industries to practice needed skills. As they become more affordable and realistic, an increasing number of mine operators are expected to consider using them (Schmidt, 2014). To assess the use of such simulations for safety skills training, researchers at NIOSH have designed and tested several simulations related to underground coal mining, including mine map reading (Mallett et al., 2009), mine rescue breathing apparatus benching (Navoyski et al., 2015) and mine emergency self-escape (Orr et al., 2009).

Many computer-based simulations are designed for self-study, allowing a single trainee to proceed through the exercise at his or her own pace, with the software providing instruction before, feedback during and assessment after the exercise. But some simulations can be delivered simultaneously to multiple trainees in a class led by a trainer.

This “classroom-with-trainer” use of simulation software is likely to be adopted where possible, as it is efficient for trainers and fits with typical training schedules. Furthermore, this use holds the promise of richer learning by enabling trainees to not only review their own actions and strategies, but also compare and contrast them with those of their colleagues during a group debrief discussion. Yet conducting such group simulation training poses distinct technical and training challenges. Therefore, NIOSH researchers wished to observe trainers using a simulation to teach groups of miners in the classroom to assess whether they had any difficulty using the simulation effectively and, if so, to provide guidance to help them use it better.

Over the summer of 2015, NIOSH observed trainers at a mine site as they used the Mine Emergency Escape Training (MEET) software to simultaneously teach small groups of trainees in the classroom. Analysis of these observations led us to conclude that trainers used the software effectively, but could benefit from some additional guidance (Connor et al., 2016). Because NIOSH is making the MEET software available to the industry (Orr, 2016), and because the guidance would apply to similar simulation exercises, a summary of NIOSH’s findings is provided here for safety practitioners using or considering using computer-based simulations to teach decision-making and problem solving.

The Mine Emergency Escape Training Exercise

MEET is a simulation designed by NIOSH to let miners develop escape skills in a realistic and interactive fashion (Orr, 2016; Orr et al., 2009). Trainees must use judgement and decision-making skills — skills that are critical for self-escape (Vaught et al., 2000; Brnich and Hall, 2013) — to successfully escape a fire in a virtual underground coal mine. MEET can accommodate multiple trainees in the same exercise, enabling trainees to interact with each other and practice escaping as a group, as they might in real life. Such interactive features have been associated with greater engagement and learning (Dickey, 2005).

The simulated environment is based on a small, three-section room-and-pillar underground coal mine. Trainees take on the role of an escaping miner, controlling that miner “avatar” by using the arrow keys to walk, the mouse to look, and the keyboard to explore and interact with the virtual environment. The screen shows the view through their avatar’s eyes (first-person perspective). As trainees use their avatar to walk through the mine, they can pick up and use self-contained self-rescuers (SCSRs), attach to escapeway lifelines, enter a refuge chamber, open and close mandoors, take gas readings, and so on. Along the way, they encounter smoke and other hazards.

MEET’s multiplayer mode allows up to four different miner avatars to be present in the same session, with one pair starting the exercise near the face and the other several breaks outby. If their avatars are in the same location, trainees can see each other’s avatars and signal each other with cap lamps or via the simulation’s text messaging system to coordinate their escape (Figure 1). Each avatar can be controlled by one or more trainees sitting around a single workstation, so that a session might involve a dozen trainees working at four different workstations. These workstations can be in the same room or different rooms, so long as they are on the same computer network.

A Study of Trainer-led MEET Exercises

In 2015, an operator of underground coal mines aware of but unexperienced with the MEET exercise asked NIOSH if the operator’s mine trainers could use it during their annual training. In return, to help us improve the exercise and further our research, we were permitted to document observations of the training sessions. All sessions were held in a single room around a table with four trainee workstations. As a result, researchers were able to easily observe the trainees and listen to them strategize with one another, react to the simulated environment, and ask questions about the exercise.

After receiving Institutional Review Board approval for the study, researchers observed roughly 500 miner trainees across 57 sessions taught by several different in-house company safety trainers. All of the trainers were new to the MEET software. To enhance research accuracy, most sessions were observed by a pair of NIOSH researchers, each moving systematically from workstation to workstation, using checklists to record trainer and trainee interactions with the technology and with each other. Once back at our facility, researchers tallied observations and analyzed written notes to answer such questions as “When did the trainees have difficulty with the exercise?” and “What did trainers do well, and what could they have done better?”

Trainee Difficulties: ‘Tech Trouble’ and ‘Escape Trouble’

Most often, trainee groups completed the escape simulation and reached safety in about 15-20 minutes. But some groups had difficulty, taking as long as 45 minutes to finish. Observers identified two distinct categories of difficulty, which is called “tech trouble” and “escape trouble.”

Tech Trouble: Helping Trainees Navigate Simulations — “Tech trouble” meant any difficulty related to the simulation, such as trainees not being able to successfully attach to the lifeline although they were standing next to it, or mistakenly donning an SCSR when they were trying to check the gas levels. “Tech trouble” robs trainees of time and attention they can devote to the scenario problem, and can lead to frustration that dampens trainees’ motivation to participate and to persist when the scenario itself is challenging (Ambrose et al., 2010). Therefore, trainers should minimize “tech trouble” as much as possible, both before an exercise through instruction and practice using the software interface and during an exercise through giving assistance.

Researchers observed “tech trouble” both before and during the exercise. The trainers effectively addressed it by giving trainees time to practice using MEET’s “preshift” practice mode, supplementing the on-screen instructions with encouragement and personalized tips, and offering individual help where needed. Some trainers advised trainees to switch seats so that all avatar teams had someone comfortable with the controls. When “tech trouble” emerged during the exercise, the trainers helped freely. This help ranged from giving a quick reminder about how to toggle some control to actually sitting down with a group for the remainder of the exercise and running all the controls in response to the trainees’ directions. In this way, trainers prevented irrelevant “tech trouble” from derailing the lesson, helping all involved in the class work through the exercise to completion.

Escape Trouble: Letting Trainees Make Mistakes — “Escape trouble” was any difficulty related to self-escape knowledge, skills, abilities, and other personal attributes (KSAOs), such as knowing how to determine one’s location and direction of movement in the mine using the tactile symbols on a lifeline or a mine map. A good problem-solving and decision-making exercise will present conditions of real-world complexity and uncertainty, coupled with realistic feedback, in which trainees may have to combine and use KSAOs in ways they never have before. Therefore, some “escape trouble” is desirable as part of the learning challenge. Indeed, it may reveal gaps in prior learning that should be discussed during the debrief and targeted with follow-up training. So MEET and simulations like it are designed precisely to create such difficulty, to let trainees practice getting out of challenging situations without risking lives or damage to property if they make a mistake.

“Escape trouble” should be handled very differently than “tech trouble.” With “tech trouble,” trainers should intervene freely, trying to remove all of it. With “escape trouble,” they should intervene very little, lest they defeat the purpose of having a simulation. Trainers need to very clearly distinguish the two types of trouble in order to take the correct approach. NIOSH observations suggest that trainers must be on guard against treating the two alike and oversimplifying the learning experience.

During the MEET sessions, NIOSH researchers rated how much the safety trainers at the mine assisted with “escape trouble.” Categories ranged from “not at all” to simply asking trainees what they were doing or telling them precisely what to do (e.g., “Go through that mandoor. You’ll be in the secondary. Just follow that all the way out.”). Researchers distinguished giving precise directions from giving a hint or a strategy (e.g, “That’s another stopping on your left. So are you headed inby or outby?”). Safety trainers were cautioned before using the software to “assist with interface issues/questions, but try to avoid assisting with response during the sim. If someone is hopelessly lost, try to get them back on track.”

Generally, trainers did not intervene much. However, when they did intervene, trainers “told trainees the answer” about as often as they merely “gave them a clue” or “asked what they were doing.” Also, sometimes the trainer intervened even when a group was not having difficulty. For example, at the start of one exercise session, a trainer announced to the class. “OK, guys, pick up your SCSRs and head for the primary.” After such an announcement, an observer cannot determine whether on their own some trainees would have chosen to escape through the returns or through the secondary escapeway, or would have either forgotten or thought it unimportant to pick up extra SCSRs on their way out.

Obviously, if trainees remain completely lost despite receiving measured assistance, then it serves no purpose to allow them to remain so. In those cases, they may need more direct guidance with explanatory commentary to successfully continue with the training session. Otherwise, trainees will learn more if trainers let them make their own choices and experience the results for themselves, then discuss what they learned during the debrief.

Computer-assisted Debriefs: Focusing on the Trainee, Not the Technology

Computer-based simulations often feature great resources to help trainers promote learning after the exercise. For example, MEET records all trainee actions in the simulation and plots them on a mine map (Figure 2), letting the trainer (or a trainee) play back and pause the action at key moments. Anything deemed a “critical error,” such as taking off an SCSR in poor air, is flagged. The playback also shows elements such as the paths that trainees took to escape and the propagation of the smoke, which helps to communicate the visual context of trainee decisions.

Such features put a wealth of data and tools in the hands of the trainers — much more than could ever be captured in tabletop simulations — but it still is up to the trainer to use them effectively. For example, the MEET data could be used to generate a simple count of critical errors, or to identify all those who took a “wrong” path. However, even more valuable is to use the data to support a follow-up discussion of trainee decision making. Especially in the case of simulation exercises like MEET, the goal of the debrief session is not to artificially reduce the complexity of the real world on which the exercise is based, but to help the trainees get better at navigating that complexity. This is best done by using discussion to draw out their reasoning — how they made sense of the information they had, exploring options not taken, and making connections between the simulation and the real world. That discussion could be followed by a formal assessment, like a short quiz. Following an educational activity with an assessment has been shown to enhance learning and retention (Haas et al., 2014; Ambrose et al., 2010).

Because debriefs come at the end of the exercise, they are easy targets for exclusion when trainers run short on time. Trainers might think running a computer-based exercise means they can cut a debrief with no negative impact on learning, because the data is recorded and can be reviewed later. But that would be a mistake, since research shows that even post-exercise debriefs that average 18 minutes in length markedly improve learning (Tannenbaum and Cerasoli, 2013, cited in Haas et al., 2014).

In the case of the invited MEET sessions held at the coal mine in 2015, the importance of the debrief seemed to be well understood. During these sessions, several unexpected schedule changes resulted in less time for the MEET sessions, which created pressure on trainers to cut the debrief short. The median debrief was 10 minutes in length: short, although roughly half the median time spent in the escape exercise (22 minutes). Nevertheless, the MEET trainers always led a debrief that encompassed a replay of the entire scenario with comments or discussion about trainee decision-making.

One area where poorly used debrief tools can exacerbate a problem emerged over the several weeks during which our observations took place. Teaching several sessions a day for weeks on end, trainers became very familiar with the typical answers trainees gave in discussion. This familiarity tempted them to cut discussion in favor of a lecture, which enabled them to finish the debrief more quickly, even when there was no pressure from the schedule to do so. This pitfall is not unique to computer-based simulations, but it is perhaps an easier trap to fall into with simulations such as MEET because the software allows trainers to simply launch the replay and let the technology drive the debrief.

To ward against this, one trainer held a discussion without referring to the computer at all. He turned off all computer monitors, moved to the center of the classroom, and started narrating what he had seen. He would bring up a decision point he observed and ask the group to explain their thinking, and then bring in other groups, or add other information. Only after covering all the main points did he launch the escape playback. This approach took more effort, but by keeping focused on the trainees, the trainer drew more participation from the class and made sure the class members could take the learning further.

Conclusion

Well-designed computer-based simulations give trainers new tools for teaching and assessing learning, which can be an advantage over tabletop and other types of simulations long used in the industry. NIOSH has been investigating the use of computer-based simulations for more than two decades, and has found them to be an effective means of developing safety-related skills and knowledge. Using research-based practices for teaching problem solving and decision-making such as those discussed above, trainers can incorporate computer-based simulations into their safety and health training activities, leveraging the advantages of the technology while avoiding any accompanying pitfalls.

To download MEET, instructions and advice for its use in multiplayer training, visit here. Contact Timothy Orr at Torr@cdc.gov for more information