Augmented Reality – 3D Navigation

Now that Curious Blocks has been submitted to the App Store, I want to step back and take a minute to talk about what we’ve learned about the control aspect of AR apps and games.

We decided last year to make a game that utilised AR, and we started experimenting with ARKit by making a number of prototypes. They ranged from local multiplayer ping pong, fighting games, to MOBA experiences. We wanted to know how AR can improve games and interactive experiences (alongside it’s limitations). 

In one of the prototypes, we decided to explore one particular aspect of AR – control. With AR, controlling the position and angle of the (virtual) camera in a 3D space all of a sudden becomes incredible easy and intuitive. The way we control the camera in 3D is standardised on many platforms; on the PC, we usually use the mouse + command/alt/control to look around, arrow/WASD keys to pan, and mouse wheel to zoom. On consoles with controllers, we use the left thumbstick to look around and the right thumbstick to pan. 

(image: navigating the 3D space in Unity) 

Then on mobile, we normally use one or two fingers to swipe across the screen to rotate, pan, and zoom. The touch gesture is fairly intuitive, however it also comes with a few limitations:

• Firstly, while we use our fingers to manipulate the camera, it stops us from using them for other interactions in the game or app. When a player is manipulating the camera, the gameplay/experience stops and waits for the manipulation to be completed before continuing. Sure, the game doesn’t have to stop – NPCs can still walk around and talk to the player. However the player cannot react or initiate an interaction since their fingers are occupied. This is less of a problem on PC/Console as there are multiple other buttons and keys that a player can still use while moving the camera. 
• Secondly, the action of swiping across the screen has the physical limitation of how far the fingers can travel on the screen, since the screen size presents a real physical size with width/length limits. This is more of a problem for panning / zooming than for rotating (since with rotation we are always dealing with a degree <= 360). 

(photo: by Lina Yatsen)

With augmented reality, we can tackle both of these issues. AR allows the player to control the camera in 3D by moving and rotating the device naturally. There is no learning curve – this is how people take photos on their phones, and this is how we move the camera in real life. As we began prototyping an AR creative app, we quickly realised the experience of creating a model in Curious Blocks actually feels like making a sculpture! When you start to move the device to find the right spots to put down your blocks, you move your arms and tilt your head to create something in-between the virtual and the real. Sometimes it even feels meditative!

For developers and 3D artists, manoeuvring the 3D space with keyboard+mouse is second nature, and it’s the same for gamers with joysticks. However, I still remember hours spent getting my hands used to the these types of controls. There’s a certain complexity whenever we map a virtual 3D environment onto a 2D screen.

(image: MagicaVoxel on PC)

With augmented reality, however, we no longer try to navigate a 3D environment via a 2D screen. The 3D environment is brought back to the physical world – which is naturally 3D. Curious Blocks may look like it’s all about stacking blocks to create 3D models, but at the core of it, it is the first aspect of AR we decided to explore and create an experience for – a _native_ 3D navigation / control.

Further more, with augmented reality, the line between what we perceive in the real-world starts to blur with what we experience on screen in the virtual space. I’ve surprised myself many times when I have reached behind the phone to try to move the blocks! Or I’ve been puzzled for a brief second wondering where the model went after I put down the phone.

Augmented reality has a lot to offer the entertainment world and we intend to continue to explore its various properties – physical, technical, and also physiological. It’s a new and exciting field and I can’t wait to share with you what we are cooking next!