Background

Piano music, and more classically-leaning music in general, is usually harder for musically un-trained populations to fully enjoy and has a high barrier of entry. This is not because they aren’t trying to enjoy it, but rather it is harder for them to hear and understand the nuances and subtleties within the music itself.

With the advent of technology like Augmented Reality (AR) that allows us to overlay another dimension on top of the world we live in, my team and I used this technology to explore the possibility of using this emerging discipline in order to make piano music more accessible and more enjoyable for people of all musical backgrounds.

UX Research

Through research, led by Ellen Nguyen and aided by Clarissa Wu, we were able to conduct interviews and surveys in order to discover how people from different backgrounds visualized piano music. The interviews and surveys included questions related to their experience with classical and non-classical music as well as listening portions where they would listen to a variety of piano pieces and then write down what they visualized when listening to them.

Originally, we had thought color would be an easy variable to adjust in accordance with the feelings they evoked; however, we soon found out that color had high emotional variability between people as it was based on their own subjective experiences. But while color may have changed too much across individuals, we also discovered that the same overall emotional response was elicited depending on what piece was being played; we also saw that the objects and environment people visualized followed common themes relating to nature. These objects included oceans, grassy fields, rivers, and trees.

Design

For design, led by me and aided by Clarissa Wu, we decided to start by taking the research insights and pivoting from focusing on different colors to brainstorming different ways that different aspects of piano music could be visualized through nature. We talked to a few music teachers and professors in order to come up with five main components of piano music that could impact our visualization; they were Tempo, Volume, Pitch, Chord Type, and Rhythm.

Then, we focused on the specific qualities of each component and tied different aspects of our design to it; some examples of this would be volume controlling the age of the vegetation, pitch controlling the location and type of growth, and chord type affecting environmental factors and colors.

Visualizations

After determining how different aspects of piano music would affect our designs, we proceeded to analyze four different variations of “Twinkle Twinkle Little Star” that I had composed already to create our visualizations for them. Motion tracking was first done with Cinema4D’s internal motion tracker on top of footage taken from an Iphone X; modeling, animation, and lighting were separately created in Cinema4D and then composited with the motion tracked footage. Rendering was done with the physical renderer and post-work, including stitching audio back in, was done through Adobe After Effects.

The first variation of the piece is a gentle, tender version of twinkle twinkle little stars. With a slower tempo, higher pitch, moderate volume, and calm major chords, this variation evoked feelings of peace, tranquility, and childlike joy. We believed that a lush grassy field gently swaying in the wind with a couple of fireflies dancing around the scene placed on the right side of the piano evoked the same sense of peace and quietness that the piece itself did; we utilized grass models from the content library and glowing spheres for the fireflies.

The second variation of the piece is a much more grandiose, celebratory version of twinkle twinkle little stars. With a moderate tempo, moderate pitch, louder volume, and full major chords, this variation evoked feelings of royalty, magnificence, and grandeur. For design, we decided that a bundle of colorful flowers and butterflies hovering around them placed around the center of the piano reflected the cheerful nature of the piece. Splines were utilized to dictate the butterflies’ paths and flower models were grabbed from the content library.

The third variation is a jazzier, more festive version of twinkle twinkle little stars. With a moderate tempo, lower pitch, moderate volume, and jazzy seventh chords, this variation evoked feelings of holiday cheer and cozy fireplace gatherings. For design, we decided that using pine trees with Christmas lights wrapped around them evoked the holiday spirit the best. I also implemented surface reflections and shadows on the back wall in this version, which turned out pretty well.

The fourth variation is the most different from the previous variations as it is a more melancholy, lonesome version of the piece. With a faster tempo, higher pitch, and minor chords, this variation evokes feelings of isolation and despondency. For design, we decided to use barren oak trees, wispy fireflies, autumnal lily pads, and a desolate park bench to complement the feelings evoked by the music.

Development

Within development, led by Woojin Ko and aided by Umang Srivastav and Edward Hwang, we were able to build a prototype of an AR application that included features such as plane detection and 3d model projection. I was able to export some models, textures, and animations into Unity for the developers to use from Cinema4D. Below is a demo video that illustrates the different features of the application.

Reflection

Lastly, we were able to conduct a few rounds of user-testing and gain some feedback into how our experience could be improved in the future. Most of the users thought the visualizations matched what they imagined in their head accurately and felt like it improved the listening experience; as for possible suggestions, users suggested that we could include non-mainstream classical pieces, fill up the piano’s breadth with more dynamic visuals, and also more variation in color and effects.

Overall this was an amazing research project to undertake; this was the first original design project I had led and I learned a lot from managing the team and also implementing my own AR designs within concept visualizations and an application. This was the largest 3D project I had undertaken at this point in time and resulted in 3000 frames rendered over 80 hours. I’m very excited to see how AR and music visualization will grow in the future and I hope I will be able to work in this intersection once again.

Special thank you to my team, the Jacobs Institute for Design Innovation, and the various mentors and professors that helped along the way.