BACKGROUND
Hearing loss has been present and have become more and more prevalent as the years go by. This issue demonstrates as a growing global challenge. In 2024, more than 450 million people worldwide experience a level of disabling hearing loss, and this number is projected to reach 700 million in just 25 years. Despite the apparent rapid growth, accessibility in the musical world for the deaf and hard of hearing (HH) remains quite limited.
We wanted to figure out solutions to allow the deaf/HH to immerse themselves in the musical experience. Our job is to innovate programs and showcase it to the public, with the intent to spread awareness and allow anyone in the world to be moved through our projects, whether or not they have any level of hearing loss.
OUR INSPIRATION
One solution that was present in the past was utilizing subtitles along with what the viewer was watching. But, a few word description of the soundtrack provided in the subtitles don't capture the emotional depth of music.
Many deaf/HH people have stated that they can experience music through feeling vibration through balloons. However, this only captures the broad beats present in the sound playing, rather than precisely grasping the tone, rhythm, dynamics, or melody. Vibrations are only effective when the person with hearing impairment is near a loud speaker, demonstrating that this solution hardly works in any other situation.
In 2015, an experiment was conducted, where both deaf and non-deaf participants were shown visual stimuli that simulated rhythmic patterns. Cross-modal plasticity was demonstrated whenthe deaf participants had their auditory cortex (normally utilized to process sound) activated in response to the visual rhythms. However, there were hardly any activity in the auditory cortices of the hearing participants.
This finding motivated us to create a real-time music visualizer to allow the deaf/HH audience to experience music to a greater degree.
VISUALIZE IN SYNC
Each of the twelve circles have a different color, and each represent a specific note from the chromatic scale. Once the program is run, the circles will move in random motion to resemble aliveness and a specific colored circle will be visible based on whether the note is played or not. The louder that specific note is, the more visible it will be. This depiction will allow the viewer to experience the variety of the musical notes offered by the music playing.
Below is a key to what each color resembles:
MOOD DETECTION
Music isn't just classified as a string of notes being played altogether, but they also carry a specific mood throughout the song. Our AI-predicting mood detection system, ran by a k-NN (k-Nearest Neighbors) model, will display a background shade based on whether the current part of the song is positive or negative. The brighter the background, the more positive, energetic and/or happy the current part of the song is. The darker the background, the more somber, tranquil, and/or melancholic the current part of the song is.
SCREENING
We have screened the music visualizer to various audiences who are part of the deaf or hard of hearing community. Our goal is to allow more deaf/HH viewers to experience music across the globe, both online and in person.
For online screening, we've posted various YouTube videos on our channel that visualizes music of different genres. Many have been requested from our viewers in the comments section. Below is an example of Turkish March by Wolfgang Amadeus Mozart visualized in real time!
Our in-person screenings currently occur across Los Angeles and is aimed towards deaf and hard of hearing audiences across all ages. Click below to learn more.
CAPTIONS TO VISUALS
Although captions have helped the deaf/HH audience understand the general idea of what kind of background music is playing, a brief phrase (e.g. "somber music", "cheerful choir", "intense violin music", etc.) doesn't truly capture the key characteristics and emotions offered by the song. We made sure to use our visualization tool as another complementary accessibility tool to allow the deaf/HH audience grasp deeper into what music is playing.
Below, we use the Game of Thrones Opening Credits as an example of how we can use our music visualizer to complement with the show currently playing.
CONTACT US
Have a question or want to discuss?
We're here to help. Reach out to us on the form on the left, or feel free to click on the link below to fill out a feedback form: