Fields is the world’s first augmented reality spatial sound creation tool. It harnesses ARKit and object persistence to easily arrange three-dimensional soundscapes with nothing more than headphones and a mobile device.
The culture of spatial sound had been stagnant. Consumers not only continue to listen to music with a flat stereo image, but the accessibility to compose in 3D is continuously left to “the professionals.”
The need for convincing audio in virtual reality, one that truly transports the viewer, has rebooted the larger conversation around 3D sound. Apple's recent developments introduced the potential to use mobile AR to anchor audio files in a user’s physical surrounding space.
I was the project manager for the small team behind Fields while wearing several other hats. UX/UI design was a major responsibility, which called for wireframes, rapid prototyping with InVision, and content creation such as UI icons, UX copy, the app’s logo, an internal style guide, and all marketing materials. I handled artist licensing and assets for the Discover section, and ran a press campaign that landed features in Pitchfork and Create Digital Music.
Fields would possibly be many users' first real interaction with augmented reality. Pokémon Go helped introduce the experience of mobile AR to a vast audience. There are a handful of apps, including GIPHY World and IKEA Place, that were at the forefront of ARKits' launch, but they were offering consumers something they were already familiar with. Fields would also be putting the capability to arrange spatial compositions in the hands of anyone, especially those with zero audio production training, for the the first time.
Fields uses object persistence to anchor a sound file in space. A “field” is an arrangement of multiple sound files in a surrounding area. The magic of the app is the sense of physicality that each sound acquires, which come to life as particle bursts on your screen. The experience is different each time, as the volume and position of each sound corresponds to your movements as you walk through them in a field.
We prioritized a basic UI so that Fields would be best experienced with the ears. The camera feed and visual representation of sound on the app would be accompanied by functions that felt familiar to iOS users, but wouldn’t distract them with flashy design.
Fields is founded on simplifying a complicated creation process, and it was crucial that the UI was discoverable. I looked at a wide range of iOS apps to draw influence on the primary navigation before designing the icons by hand.
We considered multiple attenuation settings (i.e. inverse, exponential, linear) in our testing to determine what worked best to create dynamic audio sensations, and dug deep into various algorithms and roll-off factors to fine tune the apps settings. It was always planned to have these exposed, but how?
The initial and core function of Fields is recording sound with the device’s built-in microphone, which anchors it on a three-dimensional plane. Planting pre-recorded audio was on the table, but Fields started with live sampling because of the instant gratification...of preserving a moment in time and in place.
The library began as an active field screen, where one could view all of the live recorded sound, and choose which ones to delete or save to the phone in the app. The active field screen broke off and became "tracks," allowing you to access your arrangement as a list. We added default sound samples to the library for mic-shy users and connected iTunes File Sharing to upload one’s own sounds, making it simple to plant them in a spatial arrangement. This sparked the rapid development of a Discover section to intrigue and inspire.
The launch of Fields featured original works by Matmos, Robert Lippok, Ami Yamasaki, Matthew Patterson Curry, and Nils Berg Cinemascope. These artists have carved out their own style of creative audio production with a deep appreciation of spatial sound. The Discover section started as an over-designed portal that included hi-resolution artist photos, condensed bios, and external links to their social media assets. These were all stripped down to their names and a title for the piece, much like a piece of art that is best represented by the work itself.
Recording videos that preserved the 3D audio experience into an iOS Camera Roll export was also one of the first core functions of the app. It allows you to immediately share your experience with anyone, even if they don’t have Fields, as long as they can receive and hear and .mov file. It’s also the sole feature in Discover, allowing listeners to record their experience but not tamper with artists’ original materials. This opens the door to Fields as an ongoing release platform for 3D works.
I was using Sketch, Craft, and InVision to constantly test the flow and assess the discoverability of every move. The style guide I created was a common ground for colors, typeface, and other design elements that offered reference. The app’s logo is, in fact, a deviation from the flat imagery of most apps - a photorealistic vintage ladybug that represents the user in a wide world of surrounding sound.
Onboarding was a high priority. Nothing like this had existed before, and Fields had to deliver augmented reality and spatial sound in a digestible format. Challenge accepted!
There were several details that needed to be made obvious, such as headphones being required as well as walking through a field - not just a rotation of the body in place.
We had to convey this in a way that matched the overall simplicity of the app’s UI. User testing was the key component, synthesizing countless demonstrations and conversations towards an enlightened user flow and copy that anticipates their needs and pain points.
My years of publicity experience came in handy to strategize the public launch of Fields. We secured an exclusive feature with Pitchfork, making it the first app to be covered by the widely popular website.
Its writer, Noah Yoo, called it "one of the coolest audio AR experiences ever," adding that "if you're a producer or a singer or if you're interested in sound and music in any way, you owe it to yourself to check this out."
Fields 1.0 was released on the App Store on May 15, 2018. An iPad compatible version was launched three days later. The latest version features new artists in Discover, and updated UI that you see throughout this page.