Augmented Reality Design – Crossing the Screen Space Divide
Also Posted On

Currently, we have plenty of great digital flat screen interfaces and fun Augmented Reality experiments, but there are few examples of user interfaces that traverse 2D and 3D spaces when efficient for a user. Typically, flat UI is constrained to flat planes and most AR objects float independently within a scene. Interfaces that can switch between 2D and 3D give us the opportunity to take advantage of both spaces.

Tied to Space

Our design mediums are grounded and tied to the world around us. Video games have been using a variety of representations to communicate within a world and it’s space. These range from diegetic heads up displays and in game UI that both the player and character can see, to non-diegetic UI that only the player can see.

MDN
This is a Diegetic heads up display from the GameCube game Metroid Prime released in 2002. Both the character and player experience the same set of information through a first person view. It would be confusing for any of the HUD information to move independently because it’s fixed to the character’s helmet view and by extension the player’s perspective.

While games bend and break the laws of nature for game mechanics and storytelling, general concepts like depth, perspective and volume are still rooted in our real world. In the same way, film carries us through 3D space with consideration to composition and perspective. ‘Graphical space’ has been explored in design with things like El Lissitzky’s Proun Room, designed in the 1920’s. Graphic, User Interface and Interaction Design often employ space and depth (along with more representational types of skeuomorphism) to communicate emotion and behavior. Even if interfaces try to avoid skeuomorphism, UI interactions and transitions are still based on behaviors grounded in physical realism.

Medium-Sized Spaces

Each medium shapes how things behave in it’s ‘world’ which in turn gives people expectations of how it should behave. William Morris, a contributor to the Arts and Crafts movement is attributed to the quote:

“You can’t have art without resistance in the material.”

Sidestepping the question of what art is, it’s clear that our mediums invoke strong forces on how we create things. Marshall McLuhan’s coining of the phrase “The medium is the message” is the classical line of thinking applied here. Mediums shape our content/form/function based both on how things fit into people’s lives (1) – as well as technical and physical contraints around how the information itself is delivered (2). As each one of these mediums (applied, we can think of them as disciplines) develop on their own they create new things and methods (sometimes radically different, beautiful things) and push against their medium.

Knocking Down Walls

As mediums develop and overlap however, distinctions between disciplines get more fuzzy (3). Augmented / Mixed Reality blur the lines, allowing objects and information to be ‘placed’ in a personal or multi user space.

AR benefits from borrowing techniques (similar to video games combining interaction, audio and visuals to tell stories) from film, UI design and game development. Naturally, design conventions and patterns (4) develop as a combination of the previous mediums in a new unique space. These disciplines increasingly overlap with AR and other spacial, context aware approaches. Here, we can continue to look at what current approaches have to offer while learning from some of the amazing work done before us.

AR: 3D Visualization

The idea of using three dimensions to provide spacial context while visualizing information is a topic that’s already been explored. Muriel Cooper (5) and her students pioneered a wide range of this work at the Visible Language Workshop in 1975 (and after it merged into the MIT Media Lab in 1985). Information Landscapes was far ahead of its time, employing new modes of interaction in 3 dimensions with data visualization and typography.

Information Landscapes. Muriel Cooper, David Small, Suguru Ishizaki, Earl Rennison, Lisa Strausfeld. (c) MIT Media Lab. July 1994.

This set of work found new ways to use space and time to show relationships between different pieces of information. Lisa Strausfeld’s Financial Viewpoints project used these strategies for visualizing large sets of data in a 3D space.

MDN MDN MDN
Lisa Strausfeld’s Financial Viewpoints visualizing 7 mutual funds.

This project allowed people to explore an information space in a way that was both reminiscent with how they explore the real world but also without physical restraints, like the effort involved with walking. Lisa writes:

“3D virtual information spaces, because they are abstract worlds of suspended text and images rather than simulations of the physical world, have a somewhat paradoxical relation to the body. As explorers of these spaces we become (at some level of consciousness) both embodied and disembodied. We rely on bodily intuition to navigate through and understand the structure and contents of the space, but we also have the ability to do what our bodies do not allow us to do in the physical world: we can fly (even through time), change our size (e.g. zoom through multiple scales of information) and see through objects (via transparency).”

Financial Viewpoints opened additional layers of information for viewing and gave people the ability to traverse that space. In an analogous way, AR allows people move in unrestricted ways through space while remaining grounded in the surroundings around them.

AR: Mapped to the Real World

Many AR apps and experiments place and create objects within a space based on their world’s representation. At the same time, being in a real space imposes boundaries. People must still avoid running into physical objects and walls, and obscuring too many of these reference points with digital elements can be disorienting. Users sometimes need to walk over to something they dropped within an AR space to manipulate or remove it (contrast this with a video game world, where movement is only as limited as the developers make it). While this encourages exploration of space and direct manipulation of things, AR objects can still be isolated without any efficient means of controlling them.

AR: Sensors

In addition to being transparent about surroundings, AR (or context based sensors in general (6)) can make use of additional signals and inputs. This opens up a range of creative applications that can utilize things like computer vision, geolocation, ambient light, user position and tangible interfaces. A number of artists, designers and programmers have explored these (7).

MDN
A collection of AR Experiments featured by Google.

These experiments combine features that open up new interfaces. Approaches like these align information and controls with how we might expect them to behave in the real world – depth / perspective, collision, and movement in space. Paper Cubes by Anna Fusté and Judith Amores allow the placement of real world cubes to create and redirect virtual objects.

MDN

These block markers (using image recognition) allow for more overlap between AR space and real space. People can adjust the blocks intuitively like they would any other physical block and see how it effects the stick figures in digital space.

Closing the Gap

Instead of separating the AR world and flatscreen UI, we can create things that move fluidly between both spaces and take advantage of both places. This may mean doing more then enabling tap/touch/point events using ray casting on 3D objects in space. Generally, objects in world space hope to be controlled / labeled. Local flat UI (which people are already familiar with) often wishes it could better reach out into worldspace to do the controlling. This creates questions around how UI layers should interact with simulated depth (on screens the representation is still ironically flat (8)). Andrew Hart’s retail and locational concepts demonstrate a combination of knowing when and where to position 2D and 3D UI. 2D isn’t inherently better or worse than 3D, there’s a time and place for both.

Interactions like throwing a Poké Ball in Pokemon Go shift from the mode of the local flat screen to the AR world space fluidly. These sorts of patterns allow us to stay grounded in the utility of existing UI conventions while also taking advantage of what AR has to offer.

To illustrate this in a different way, I designed a ‘push to world’ UI that switches between 2D and 3D modes of showing live seismic earthquake data from the United States Geological Survey. The ‘zooming out’ of the panel aims to communicate that the UI window has shifted over to a new space and behaves differently. Joni Korpi’s modeless zoomable UI concepts and patterns function in a similar way here – space and transitions are used give people more context. Joni writes about this in his Zoomable UI for the Web article:

“Every article feels like it has a spatial location on the site. The zooming animation helps me understand where elements come from and where they’re going. I’m trying to aim for a system that makes me think ‘the article with yellow bars somewhere in the bottom left” instead of “the article named Golden Grid System’.”

Animation (and having 3D space in general) can help people with mental organization. A X/Y/Z space offers the opportunity to use real ‘depth’ in our 3D UI.

Testing a the 2D > 3D interface

In the same spirit as Muriel Cooper’s Information Landscapes, this depth gives us a convenient option to look at information from another angle. In this case, showing data in 3D can give us a better sense of scale and proportion (seismic waves are measured on a logarithmic scale).

A proof of concept for a UI pattern that presents real time seismic earthquake data in both 2-dimensional and 3-dimensional modes. Note how the interface transitions from 2D local space to 3D world space in order to show seismic information from a new angle.

Being able to see depth and perspective helps us gain an appreciation for differences in magnitude relative to other earthquake sites. While the same set of information could be visualized in other ways, a 3D representation lets us see details that might otherwise be lost or take up too much space in 2D. Conversely, flat graphics are a more efficient way to communicate and traverse simple sets of information. The game Subnautica allows the player to manipulate and hold onto (or fire not so gently) objects in space, partially bypassing problems of distance.

MDN
The Propulsion Cannon in the video game Subnautica

Maybe we can apply similar techniques like these to manipulating UI windows (9), allowing them to be quickly tethered and retrieved to local flat UI. There are benefits to both (10) 3D and flat space, and methods for fluidly transitioning in and outside of those modes.

Staying Rooted

Digital mediums will continue to become more in sync with our world, giving us the opportunity to help define when and where information shows up. As we do this we can explore how interfaces shift and adapt to people and their space’s needs, virtual or otherwise (the medium). There’s a lot to learn from the progression of both 2D and 3D disciplines. As we string together flat rectangles with spacial boxes we should make sure we’re anchored to the right places along the way.


Questions or comments? DM me on Twitter or email me!


Footnotes

  1. Vernacular architecture is based on solving for local needs and draws from nearby materials and traditions (thanks to John Ratajczak for noting this topic).

  2. Paul Ford’s interview on Scratching the Surface goes into detail about how the medium of the web shapes its content.

  3. An article on the effects of Hyperreality – or “inability of consciousness to distinguish reality from a simulation of reality” dives into this. While the article itself delves into the topic on a much wider social and economic scale, we can note the points on how the medium can be bypassed in the creation of a message. Jean Baudrillard’s Simulacra and Simulation is referenced in the article author stating:

“Baudrillard means the ‘zero degree’ of meaning, the triumph of form over substance, a ‘simplified operational mode’, a seductive and pseudo-consensual presentation. Things are presented in such a way that the surface effaces and covers up any possible depth.”

Current examples might be something like high quality renders of locations that appear to be photographs or recorded video, or objects placed in AR spaces that take on similar attributes (material properties like reflectiveness or luminosity) to real world objects.

  1. There are a number of guidelines around designing for AR, including this Prototypr Post by Alexandra Olarnyk and Google’s AR design guidelines.

  2. More on Muriel Cooper (1925–1994) and her work can be found here and in a book celebrating her career by David Reinfurt and Robert Wiesenberger.

  3. Potential sensors and signals go far beyond cameras and computer vision (see Ubiquitous computing / Internet of things). This also opens up a range of serious questions about privacy). Interactive and installation art have long used sensors to gain some element of responsiveness. A non invasive example would be Maurizio Bolognini’s Collective Intelligence Machines series which allowed people to send additional inputs through their phones. Peder Norrby’s more recent experiment Parallax View uses the iPhone X’s face tracking technology simulate depth on the phone.

  4. Some artists/designers/programmers that have explored exciting things in AR include (but are far from limited too): Andrew Hart, Anna Fusté, Judith Amores, Keiichi Matsuda, Vishnu Ganti and Zach Lieberman. The topic as a whole also has plenty of accompanying research since VPL.

  5. Escaping the screen, Joanie Lemercier’s 3D projection experiments demonstrate volumetric imagery and UI using projections through a water mist medium.

  6. It’s important to note our interfaces are tied to hardware. Interface conventions will adapt as vehicles (be it headgear, or a volumetric medium) for AR become established or rejected.

  7. Local and virtual worldspace aren’t always completely distinct. Concepts like Lee Vermeulen’s work use real world screens to show an AR camera’s view lined up in world space. Approaches like these allow for further mixing of real and virtual world space.


Other independant blog articles to check out