Mixed Reality Could Be the Future of High-Tech Coaching in Team Sports

Every pitcher believes he has enough gas for one more strike-out, but an invention from NJIT Associate Professor Tao Han might forever change that old stare-down between players and coaches on the mound.

A coach wearing a mixed-reality headset could receive real-time statistics and facial interpretation data to evaluate whether the pitcher is right to stand his ground, what he should change or whether it’s time to call the bullpen, while also interfacing with a team of analysts and trainers who work remotely and see the same information.

As today’s baseball analytics are called sabermetrics, then one might say Han’s high-tech version, applicable to any sport, is a lightsaber. He calls it holographic telepresence and obtained a National Science Foundation Innovation Corps grant to commercialize the technology, which he calls a holographic telepresense system. NJIT is a key participant in the NSF I-Corps program.

In explaining why it’s important to bring mixed reality and telepresence to sports, “We use much more Zoom and Webex than before, and more and more companies adopt hybrid working modes [such as] three days a week in the office and two days at home. But, until now, we only have 2D online environments which are less efficient in engaging people as compared to in-person,” Han noted.

“Sports training and coaching is the first area that we want to explore because 3D holography can be significantly better than 2D in terms of remotely training players and visualizing and analyzing players’ status … In addition, the metaverse is a hot topic and many companies will start to invest in this area. Our telepresence solution can be a key technology that connects the physical world and metaverse.”

There is already plenty of technology in professional sports, such as nearly ubiquitous cameras, some of which might be used for cheating. Han is focusing on the positives. “I looked at a lot of applications of holography and augmented reality/virtual reality. I think sports is a very good area and they really need the technology,” he said.

“We have talked to many coaches for basketball and for soccer, even football. I think they like the idea. … They like to understand the status of the players. They want to have these visual interfaces. If they wear the glasses, they can see all this data from the player. The data is structured and visualized so they can understand what training the player has taken and what is their status.”

Han’s assistants in the Ubiquitous Networking and Intelligent Computing Systems Laboratory are Ph.D. students Yongjie Guan, working on the camera system, Xueyu Hou, helping with machine learning algorithms and Pedro Regalado on the holographic software portion.

But equally important is research assistant Tyler Stafflinger, studying for his master’s degree in electrical and computer engineering. Stafflinger was a star pitcher for the NJIT Highlanders baseball team who emerged as a postseason surprise in 2021. He is also currently a graduate assistant for that team, so he appreciates Han’s ideas more than most ordinary engineers ever could.

Stafflinger, of Manalapan, said he likes the research because it lets athletes and coaches be more proactive instead of only reactive.

“I wanted to do more of a predictive look at the sport in general, using both video, statistics and pitch data, to do a more predictive approach. I want to be able to predict the next pitch that will be thrown and see patterns,” he explained. “People are getting more into the spin rates, the spin axis and everything. I don’t think it’s too far off from really being possible to predict with high accuracy what pitches they’ll be throwing next, because humans are habitual.”

“My final goal is to use it in either a predictive way for how batters are going to be approached, or how pitchers should approach the batters,” Stafflinger added. “You have so much data that you can use. I’ve seen it now, having coached a little bit. Sometimes it’s not always the best to use that. But more often than not the data backs up what you should be doing. You’re just looking for those tiny advantages.”

With other possible applications ranging from healthcare to the metaverse, “If you can do anything that you’re really passionate about and interested in, it makes it that much more enjoyable,” Stafflinger noted. “If we are able to be successful, then this could be a big push forward not only in sports but anything else.”