Making music requires a closed feedback loop to evaluate the created with the intended sound. This becomes a challenging task for those with hearing disabilities who are nevertheless interested in learning to play an instrument. Due to limited access to the auditory channel, deaf people have less information to evaluate their performance. The result is a feedback loop-gap.

We developed Music Sensory Substitution (MuSS) Bits, small wearable plug-and-play sensor-display pairs that communicate wirelessly peer-to-peer. The Sensory-Bit captures real-world sounds from various audio sources, extracts rhythm information and sends it to the Display-Bit which provides a visual and vibrotactile representation of the rhythm. The vibrotactile intensity and visual brightness can be adjusted to the user’s need. The decoupling of the display from the sensor part of the sensory-substitution system allows the performer to spatially deploy the Display-Bit, for example on his or her body, provides free limb movement and permits a simple way of selecting the desired audio source. Real-time feedback has been established through onboard processing, nRF communication, and responsive vibration motors.

We believe that MuSS-Bits are a first step towards customizable exploratory music-sensory-substitution systems for deaf people.


Steve Howard Award - for the best long paper by a student presented during OzCHI, CHISIG’s Annual Conference: MuSS-Bits: Sensor-Display Blocks for Deaf People to Explore Musical Sounds. December 2016


Benjamin Petry - Oral Defence

Video of my oral defence about MuSS-Bits.

MuSS-Bits: Music-Sensory-Substitution Bits for People with Hearing-Impairments

If you have the desire to make music, you should able to do so.


  • Benjamin Petry. 2018. Design and Evaluation of Music Sensory-Substitution Systems that Support People with Hearing Disabilities to Make Music. Doctoral Dissertation. Singapore University of Technology and Design, Singapore, January 2018. [Video] [PDF]
  • Benjamin Petry, Thavishi Illandara, Don Samitha Elvitigala, and Suranga Nanayakkara. 2018. Supporting Rhythm Activities of Deaf Children using Music-Sensory-Substitution Systems. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). ACM, New York, NY, USA, 486:1–486:10. [Link] [DOI]
  • Benjamin Petry, Juan Pablo Forero, and Suranga Nanayakkara. Muss-bits – music-sensory-substitution bits. 2016. Singapore Provisional Patent Application: 10201610020P. Filing Date: 29 November 2016.
  • Benjamin Petry, Thavishi Illandara, and Suranga Nanayakkara. 2016. MuSS-Bits: Sensor-Display Blocks for Deaf People to Explore Musical Sounds. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI '16). ACM, New York, NY, USA, 72-80. [PDF] [DOI]
  • Benjamin Petry, Thavishi Illandara, Juan Pablo Forero, and Suranga Nanayakkara. 2016. Ad-Hoc Access to Musical Sound for Deaf Individuals. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16). ACM, New York, NY, USA, 285-286. [PDF] [DOI] [Poster]
  • Benjamin Petry, Jochen Huber, and Suranga Nanayakkara. 2018. Scaffolding the Music Listening and Music Making Experience for the Deaf. In Assistive Augmentation. Springer, Singapore, 23-48. Cognitive Science and Technology Series. [Link]