In an inviting space full of vibrant bold colors, fiber optic curtains, and a vibrating haptic chair, sounds of “Row Row Row Your Boat” and other popular children’s songs fill the air, and children with autism are becoming their own composers, learning to love music.
This is the scene in the Sensory Room at the Route 9 Library and Innovation Center, where the music is theirs to alter as they see fit. When children like what they hear, they pause to listen more closely, smile, or dance. Other children focus intently as they explore the many combinations of sound available at their fingertips. Some young listeners take delight in adding a drumbeat or fast countermelody while others seem to prefer a calmer rendition of a familiar tune. As they listen, these children learn what they like to listen to and what they don’t, providing a valuable glimpse into how they respond to musical sounds.
The children are piloting a listening device developed by University of Delaware researchers Daniel Stevens, a professor of music theory in the School of Music within the College of Arts and Sciences, Matthew Mauriello, assistant professor of computer and information sciences in the College of Engineering, and their respective students.
The professors’ divergent backgrounds were a complementary match for this innovative project that aims to better the lives of children with developmental disabilities. Together, they applied for and were awarded $50,000 from the Maggie E. Neumann Health Sciences Research Fund to advance their research. The fund specifically targets interdisciplinary research and innovation that aims to improve the lives of people with disabilities.
The device is the dream of sophomore Elise Ruggiero, a double major in music performance and psychology. Her younger brother was diagnosed with autism at age 2.
“I started playing violin at age 9. As I advanced in the music field and had recitals, I noticed that sitting still and listening to music was a challenge for my brother,” Ruggiero said. “When we’d go out to eat, if the restaurant was playing music too loudly, it would make him extremely anxious, and there wasn’t much we could do about it.”
In a freshman honors music theory class, Stevens tasked his students with solving a problem in the community.
“I asked students: ‘How would you like to change the world in which you live and work with your music skills?’ My challenge was met with stunned silence,” Stevens recalled.
But students quickly got to work, reaching out to local organizations, identifying issues, and dreaming up ways to solve problems. Ruggiero used her personal experience to team up with Autism Delaware, and her idea to create an interactive music device for children with autism was ultimately selected to move forward as the class project.
“It was really satisfying knowing that something I knew was a problem I wanted to tackle for so long is achievable,” Ruggiero said. “Seeing other people who are passionate about it too made me realize that together we can make a difference.”
Music theory students in Stevens’ class spent hours designing various renditions of what the team has been describing as modular music that’s modifiable to suit a child’s listening needs and preferences.
“Listeners with autism have real needs. Those with auditory sensitivities, for example, may be unable to participate in the formative experiences that children have singing songs with their parents or classmates, in part, because the music might be too fast, or it might have too much stimulation, or it might not have enough stimulation,” Stevens said. “Every child with autism is different, so we need to compose music that would address various needs.”
Had a device like this existed years ago, Ruggiero said it could have helped her brother.
“He was turned off by the idea of making music at a young age because he was so sensitive to sound,” Ruggiero said. “For other kids with autism, I want them to have the option to want to make music.”
Mauriello joined the project shortly after its inception to help design, build and deploy the technology in the field. He’s passionate about applying computing to challenges related to social good using his background in human-computer interaction, a blend of computer science and engineering, design, and psychology.
“I enjoy opportunities to understand and empathize with users. This allows me to build technologies that meet their specific needs,” Mauriello said.
With generous support from the Maggie E. Neumann Health Sciences Research Fund, the researchers transformed an idea into a prototype. Now, a controller housed inside a white 3D-printed box with a series of presets, or light-up buttons with pictures of instruments provides a potentially infinite amount of sound combinations and aims to enhance the listening experience for children with autism. Every time a child presses a button, the sound or melody changes, sometimes slightly, other times dramatically; each interaction is recorded so Stevens and Mauriello can gather data about listening preferences and find new ways to display this data back to composers to help them create more suitable music.
“We want to understand the way children with autism hear the world and interact with music by looking at the larger patterns that start to emerge in the data,” Stevens said. “Music is such a rich artform, and yet we hear it so frequently, we take for granted melody, harmony, texture, rhythm and all these elements that work together to make every listening experience enjoyable. When it comes to listeners with autism, every sound is up for grabs. It’s been really rewarding to think about how music can serve the listener. The needs of this particular group of listeners invite us to think creatively about how sounds can be manipulated and designed to meet their needs.”
That’s an area of particular interest to Simon Brugel. The sophomore computer science major, who’s on the spectrum, brings personal experience to the project. He said he is sensitive to loud noises.
“I don’t like squeaking or alarms,” Brugel said. “I can notice some subtle sounds others might not notice, and I prefer some instruments over others.”
Brugel helped design and write the software for the prototype and never expected to work on a project with potential for broad impact this early in his college career.
“It’s satisfying to know that my creations are having an impact on the community or the advancement of research,” Brugel said.
By participating in this interdisciplinary research, Mauriello wants his students to understand that computing technology can serve diverse populations.
“To help broaden participation in computing, we need to demonstrate that computing can have an impact on diverse problems that are facing society,” Mauriello said. “This project offers a nice opportunity for that as computer science and engineering students work with music students to build something that can have a real impact on the world.”
Abby Von Ohlen, a sophomore music education major, loved playing a role in this project and watching the idea blossom.
“Seeing this idea come to fruition has been such a good experience,” Von Ohlen said. “I’ve always been able to enjoy music and not be overstimulated by it. It’s interesting to see that even just changing one track or sound level can affect someone. It’s fulfilling to know that others will be able to enjoy music as much as I do.”
Ruggiero has observed initial trials for the device and said feedback has shown the device can be engaging and might be more attractive to children if it looked more like a toy.
“A parent of one of the children suggested that he might enjoy the device more if it was shaped like a fire truck that they could wheel around while listening to music,” Ruggiero said. “If it was more physically appealing, it might make kids more inclined to play with it.”
For older children, Ruggiero envisions an app being useful.
“If a teen or adult is out in public and something bothers them, they can modify it or use their own music on their phone to calm themselves, I would love that,” she said.
Through working on this project, Ruggiero got a lot more than she ever dreamed of in her first year of college. She had simply hoped to meet new friends and become well-adjusted to college life.
“I was not expecting to have my idea go as far as it’s gone. It makes me so happy and excited,” she said.
Now, she’s dreaming of a career in music therapy.
“This project made me interested in the research aspects of music and psychology,” she said. “I want to work with people on the spectrum and make music more accessible to them.”
Ultimately, Mauriello and Stevens said they hope the music listening device becomes a permanent fixture in the Route 9 Library’s sensory room. They also hope to incorporate the device in music and special education classes.
“The research is very clear — music participation is incredibly important to a child’s social and emotional formation, their motor development, and their interactions with family members, other children and their community,” Stevens said. “We’re inspired to make formative, engaging, participatory musical experiences accessible to every child with autism in our state and beyond over time.”
For more information on the project, email asdmusic.ud@gmail.com.
About the fund
Maggie E. Neumann Health Sciences Research Fund was established in 2020 to support research designed to improve health and quality of life outcomes for children and adults with physical and developmental disabilities. While the fund resides at the College of Health Sciences, the intent is to support interdisciplinary research across all colleges.
The research fund was created with a gift from Donald J. Puglisi and Marichu C. Valencia in honor of their granddaughter, Maggie E. Neumann. Puglisi is a member of UD’s Board of Trustees and they both serve on the President’s Leadership Council.
