You could call it “Logical progression” after my first steps into VR with cardboard, it was time to try the Apple ARkit. Again like the cardboard experience, a low-cost XR experience on mobile.
The basic requirement for any AR experience is the ability to create and track a correspondence between the real-world space and a virtual space where you can model visual content. When your app displays that content together with a live camera image, the user experiences augmented reality: the illusion that your virtual content is part of the real world.
Augmentation is not only visual, what about sound?
Time to try this out. First, I wanted to find out the basics like placing an object with real-world dimensions. (video 1) Second, find out how we can use sound in AR. (video 2) Third, combining it all, visual, audio and location (ARKit + GPS data) (images) and the last video is an experiment using ARKit without the visual camera feedback, controlling sound by moving the device.
First impressions:
– ARKit develops a better understanding of the scene if the device is moving, even if the device moves only subtle.
– Allow time for plane detection to produce clear results.
ARKit and GPS data
During the test process, I came up with the idea to combine the local ARKit tracking with GPS wayfinding. This resulted in a rough working version of a “Pokémon Go” type of experience.
So I came up with a test case app with the following objectives:
– play specific parts of music at a specific location in the real world
– visualize “the presence of sound”
– show it on a map
To create the maps I used the developer tools from Mapzen. Unfortunately by the time I’m writing this post, Mapzen is shutting down its services.
You can see the outcome of these tests below.
Digital content & experiences can become scarce or exclusive again, unable to be infinitely copied and possibly more valuable as a result, as it can only be experienced in a physical location (and maybe at a single time).
Similar to the way Live concerts are more valuable than recordings.
This motivates me to keep the creative process going, pushing the boundaries of music culture.
First tests with AR kit:
Drum Machine on desk
Tests AR kit and GPS Data on iPhone 6s:
Experiment at Middelheim park and random location in Antwerp.
Specific sound at specific location, and how do we visualise this?
Out of the box: using ARKit without camera feedback. Conclusion: the Google SDK does a better job for this.