Adapting to every (virtual) scenario
The Mines Police Department is always ready to protect and serve the Oredigger community, but when an alum called on them to test his new augmented reality training platform for first responders, they were ready for that, too. Â
Mik Bertolli â€™06, MS â€™07, chief science officer of Avrio Analytics, reached out to Dustin Olson, Minesâ€™ Director of Public Safety and Chief of Police, with an idea: He thought the AR training Avrio had developed could be used by police officers, but he needed help testing it for this new purpose. Would the campus police department help?
Bertolli brought one of Avrioâ€™s augmented reality-enabled headsets by for Chief Olson to try. â€œBasically, it was a chemical spill, the scenario I was observing,â€ Olson said. â€œAnd then Mik talked about real facilities and how he can bring that into the technology.â€
Wearing the headsets, Mines police officers would be able to run any training scenario happening in any building on campusâ€”whether they are physically in that building or not. The platform has the ability to recreate any space on campus. â€œOne of the cool things we offer on the platform is to have a tabletop of the actual building, on the actual Mines campus, and then go directly into an active shooter or de-escalation drill thatâ€™s maintaining context,â€ Bertolli said.Â
But with an active-shooter training session, for example, a police officer can still see through the headset to the room around them. The platform then augments it, adding in virtual people they can interact with and making real-time changes. â€œAs the officer is talking, the character actually responds to what theyâ€™re saying,â€ Bertolli said. An instructor with a tablet monitors the officer and can ramp up or scale back the virtual charactersâ€™ aggression levels, but the training proceeds automatically. â€œThereâ€™s a lot of machine learning that happens back there,â€ he said.
This was a marked difference in how police officers typically trained, Olson said. â€œWhatâ€™s way more common is, you have a fixed room in a police training facility and inside youâ€™ll have 180 degrees of screens, and then the participant interacts with the screen and there are scenarios,â€ Olson said. â€œThereâ€™s an instructor there that runs the scenario and grades the participants.â€Â
Officers had to go to the training facility, usually at a police academy, sometimes in another city or state. â€œItâ€™s better than nothing, itâ€™s pretty good,â€ Olson said of the traditional training. â€œBut that said, we can make this way more dynamic, and we can use facilities and the environment youâ€™re most likely going to operate in.â€
Over the next year, as Olson and other Mines officers began testing the headsets and scenarios Bertolli and his team programmed, they honed in on their de-escalation training needs. â€œWeâ€™re dealing with peopleâ€”thatâ€™s our job,â€ Olson said. He explained some of the feedback they gave that would help Bertolli create conversation with a first responder. â€œLetâ€™s say someoneâ€™s dealing with a mental health crisis.â€ Olson said. â€œMik would ask, â€˜whatâ€™s that like, whatâ€™s the dialogue you might encounter as an officer?â€™ Or in another case, maybe someone is really agitated to see you, theyâ€™re combative, potentially theyâ€™re holding a weapon.â€Â
In addition to creating locations and infinite scenarios through machine learning, Avrioâ€™s headsets give performance feedback via biometric tracking, following the userâ€™s eye, head and hand movements. They can also track heart rate, oxygen levels and galvanic skinÂ
responseâ€”in short, whether youâ€™re sweating. Collecting this data lets an instructor and the platform itself give users specific, fine-tuned feedback. â€œWeâ€™re able to tell them things like, â€˜you never looked at their hands,â€™ or â€˜you only had your eyes on the subject for 10 seconds, so were you distracted?â€™â€ Bertolli said. He added that this data lets them calculate the userâ€™s cognitive loadâ€”how stressed someone was during the training. â€œWeâ€™re able to get to nearly 90 percent accuracy on how mentally fatigued you are,â€ he said.
Before developing this AR platform, Bertolli and his colleagues had been using machine learning to analyze training data for the federal government. But when he started exploring the vast amounts of personalized data that the sensors on augmented reality headsets could collect, he said that as a scientist, he saw a lot of potential for improving training for first responders.Â
Now, Avrioâ€™s training platform is being used by campus public safety departments at other universities and by metropolitan police departments around the country. For police, they offer three core types of training scenarios: active shooter, de-escalation and tabletop, which lets officers across departments and municipal authorities train for an event that requires high-level planning. With the latter, Bertolli said he hopes the remote capabilities of their platform will make it easy for a chief of police, a fire chief and a mayor, for example, to conduct a training for an emergency event without even leaving their offices.
â€œNot only are we super happy that Chief Olson and his officers could help us with it, but as an alum, Iâ€™m extremely happy to see them adopting it and using it,â€ Bertolli said. â€œOur hope as a company is that training will happen more often, and I wanted it to be better, and to do that, it has to be accessible.â€