Exposing children to music & musical instruments through AR

OVERVIEW

It is an Augmented Reality app that focuses on early childhood education using music instruments. Music is a wonderful tool for children because it covers the five main areas of child development — cognitive development, motor, language, literacy, and social-emotional skills. We believe that exposing children to music and musical instruments through AR will provide benefits far beyond the traditional ways.



HOW DOES IT WORK

The program consists of four musical instruments handled by four image targets. Each instrument has its own 3D model and a track from a children’s song. Once the image target for a specific instrument is detected, the track will start playing. The program is stackable which means that the user can add on instruments as the song progresses. This program works perfectly with AR because it is interactive and allows for the user to decide what instrument they want to be played.


COLLABORATION

Sam Gonzalez


ROLE

Research
AR development
Visual design
Product design


TIMELINE

3 Week Sprint
Apr 2020


TECHNOLOGY

Adobe Illustrator
Unity
C#



DESIGN ELEMENTS


CODE SNIPPET


DEVELOPMENT

It is a marker-based AR app. The most important part of the development process was to figure how to detect when an image target is in view. All four audio tracks are attached to their own image targets and are all in sync and looped but muted. And when the image target is detected, we want the program to unmute the audio track of the instrument in the image target.
We wanted to make sure that even when the music is not ‘playing’ (meaning even when the instrument is not shown in view), the music is still in sync so it doesn’t break the melody. This is mainly because we want our users to have the opportunity to ‘play’ the instruments and create their own unique musical experience as seamlessly as possible.



CHALLENGES

We figured out pretty easily how to get the sound from 4 different tracks, play and start on mute. The main problem that we encountered was not in the code but in the mp3 files. We recorded 4 different tracks for each instrument. The problem was not all the tracks were the same length (there was empty noise space at the end of the piano track!) On the first run there doesn't seem to be any issue. However, when the song loops you would hear syncing issues. This was solved by making every mp3 the exact same length down to the millisecond.



DEMO


NEXT ITERATION
  • On-boarding interaction
  • Add an on-screen interface to select from multiple children's song selection
  • Reposition the instruments from the image targets and add an overlay on top of the image targets so it's hidden
  • Gamification: musical quizes