Khoury News
Can’t get a private coach? This AI tool could help democratize skill learning
Backed by a new three-year NSF grant, researchers Lorenzo Torresani, Kristen Grauman, and Amy Pavel are seeking to leverage advances in AI video understanding to make interactive coaching available to all.
Hiring a private coach is a surefire way to improve your golf game or your salsa dancing, but not everyone has the means to do so. Could AI be used to bridge this gap between expert coaches and beginners, and make skill learning more widely accessible?
That’s the question that Khoury Professor and Joseph E. Aoun Chair Lorenzo Torresani is aiming to address. Along with collaborators Kristen Grauman at UT Austin and Amy Pavel at UC Berkeley, he recently received a National Science Foundation award for the proposed Skill Assessment and Guidance Engine (SAGE), an AI system that will act like a personal coach to provide guided skill learning.
The goal is to train SAGE on real-world coaching and replicate useful feedback for the user. The user would record themselves taking a golf swing, for example, then SAGE would analyze the video and make suggestions to improve technique. Recent advancements in computer vision and AI video understanding have made this possible, but there is still a long way to go.
“Preliminary experiments show that there’s a capability gap. AI systems today can describe well what a person is doing, but they don’t do well in terms of picking up the subtle differences in executions that separate, say, an expert from a beginner. That’s where this research comes in,” Torresani says.
READ: As his awards list grows, Lorenzo Torresani crafts AI assistants for help with everyday tasks
Among the project’s challenges, there is a lack of established datasets for training or evaluating AI coaching. Torresani plans to create such a dataset using publicly available online video and discussion. He also plans to pay participants to perform these activities while having expert coaches review their performance in real time.
“We call them Wizard of Oz sessions,” Torresani says. “Videos are going to be streamed to the expert coaches, and they’ll provide feedback like the AI would. We’ll be able to measure the impact of the feedback because the expert will tell you what you did wrong and then you’ll have a chance to repeat the action right on the spot.”
While there are plenty of instructional videos online, it’s one thing to listen to good advice, and another to incorporate personalized advice in a way that leads to better outcomes.
“We have statistics that indicate about 51% of YouTube users use its videos for learning new things, and YouTube by itself reaches 85% of US adults,” Torresani says. “But there is this disconnect. Watching someone perform an activity doesn’t really tell you how to do it, especially if you’re a beginner.”
Torresani and UT Austin’s Grauman, who have collaborated many times before, will work together on the modeling, data, and evaluation methods behind the coaching system. Torresani notes that there are many valid ways to coach the same swing, which makes evaluating an AI coach unusually difficult.
“Two excellent coaches may say completely different things about the same swing, and both could be right,” Torresani explains.
As such, he plans to measure success based on whether the learner actually improves. Doing so at scale requires new evaluation methods the team is developing, since asking a person to repeat a golf swing thousands of times isn’t practical.
As public education sees funding cuts for physical and extracurricular activities, it becomes more important than ever to make skill learning accessible.
“Skill acquisition is a lifelong necessity. It spans practical, creative, and physical domains, and it has proven benefits for mental health and long-term economic outcomes,” Torresani says.
To make those benefits as accessible as possible, the researchers plan to build SAGE with the direct involvement of blind and low-vision learners, with UC Berkeley’s Pavel leading the work. Too often new technologies treat this demographic as an afterthought and add accessibility features near the end of product development, but SAGE will be co-designed with blind learners from its inception.
“The online video fitness boom has essentially excluded blind or low-vision users because they can’t learn these types of skills from a video,” Torresani says. “So, we decided to develop a system that will specifically target visually impaired people, and this design will be embedded into the system from day one.”
Torresani believes in the capacity for computer vision and AI models to improve people’s lives, especially for those who are disabled or disadvantaged, but at the same time, he understands that there is no substitute for the real thing.
“We are not building this to replace human coaches,” he acknowledges. “Human experts are the benchmark we are trying to approach, but we’re not expecting to beat that benchmark. The objective is to make forms of interactive coaching available to everyone, to democratize AI coaching with the understanding that it will not be perfect, but it will hopefully enable people that simply don’t have access to human coaches.”
The Khoury Network: Be in the know
Subscribe now to our monthly newsletter for the latest stories and achievements of our students and faculty