Exposing children to music & musical instruments through AR


It is an Augmented Reality app that focuses on early childhood education using music instruments. Music is a wonderful tool for children because it covers the five main areas of child development — cognitive development, motor, language, literacy, and social-emotional skills. We believe that exposing children to music and musical instruments through AR will provide benefits far beyond the traditional ways.


The program consists of four musical instruments handled by four image targets. Each instrument has its own 3D model and a track from a children’s song. Once the image target for a specific instrument is detected, the track will start playing. The program is stackable which means that the user can add on instruments as the song progresses. This program works perfectly with AR because it is interactive and allows for the user to decide what instrument they want to be played.


Sam Gonzalez


AR development
Visual design
Product design


3 Week Sprint
Apr 2020


Adobe Illustrator




It is a marker-based AR app. The most important part of the development process was to figure how to detect when an image target is in view. All four audio tracks are attached to their own image targets and are all in sync and looped but muted. And when the image target is detected, we want the program to unmute the audio track of the instrument in the image target.
We wanted to make sure that even when the music is not ‘playing’ (meaning even when the instrument is not shown in view), the music is still in sync so it doesn’t break the melody. This is mainly because we want our users to have the opportunity to ‘play’ the instruments and create their own unique musical experience as seamlessly as possible.


We figured out pretty easily how to get the sound from 4 different tracks, play and start on mute. The main problem that we encountered was not in the code but in the mp3 files. We recorded 4 different tracks for each instrument. The problem was not all the tracks were the same length (there was empty noise space at the end of the piano track!) On the first run there doesn't seem to be any issue. However, when the song loops you would hear syncing issues. This was solved by making every mp3 the exact same length down to the millisecond.


  • On-boarding interaction
  • Add an on-screen interface to select from multiple children's song selection
  • Reposition the instruments from the image targets and add an overlay on top of the image targets so it's hidden
  • Gamification: musical quizes