How many visual interfaces can be made?

Designers spend a significant portion of their time deliberating over what set of UI patterns are most appropriate – navigating requirements, taste, and usability. It seems, though, that within a given medium, there is a finite set of physical representations that bring about a usable graphical interface.

$I = \{\mathit{leftmenu}, \mathit{rightmenu}, \mathit{topmenu}, \ldots\}$

When we start considering context – environment, information architecture, and product goals – this space grows, and the set of all things that can be communicated is incalculably larger still ($I \subset C$). But all things that are generally usable (a subset of human factors) on a screen might be completely enumerable.

There are some strong, historically consistent constraints that bound what is useful.

Perceptual Constraints

Fundamentally, humans have relied on flat images and representations to efficiently convey information since prehistory ⁽¹⁾. Our visual systems are efficient for the vast majority of information input and we’ll rely on this modality for quite some time. Our perceptual systems, which have not fundamentally changed, frame what’s practical ⁽²⁾. We can only perceive so much (distance, field of view, attention, color spectrum, spatial frequency, and detail/difference and so forth). While GUIs pack a multitude of detail, they are ultimately limited by the medium’s resolution and what we can make out – eventually hitting diminishing returns around just-noticeable difference.

Naturally, read/writable displays that confine representation to a flat plane have dominated (cave paintings, scrolls, the codex, signage, phones, and so on), followed by a longer tail of other media and modalities. Even volumetric displays (airplane control panels, VR/AR and holographic approaches) confine the majority of their dense information to flat planes.

Representation

For an interface ⁽³⁾ – different types of tasks and information are best visually suited to specific formats and media (Information Design / Information Architecture). This often involves distilling down a complex set of narratives or use cases (often with subtle tradeoffs) to articulate ideas and content well.

For icons and UX patterns in digital displays, skeuomorphism borrows visual properties of physical objects to aid people’s mental model of how things work (part of the set of all useful things that can be communicated ($M \subset C$)).

Given this, there is a limited amount of visual treatments and styles that facilitate clarity and usability. While we can generate any interface through AI, ideal UI legibility and placement still exist (e.g. color contrast and swipe zones for capacitive screens, respectively).

Interfaces are intentional abstractions of how the world works. Neumorphism and Liquid Glass embody design systems reaching the limit of their functional set of options and overflowing into the stylistic/ornamental.

Interaction

Interaction and motion allow us to communicate additional behavior. Things like weight, attractive forces, and depth can be afforded as people manipulate UI. Google’s Material Design famously modeled shadows after the depth of stacked paper (appearance and interaction). Liquid glass is another permutation of this through real-time caustics simulation and through IMUs.

There is also a limited set of human interaction metaphors and models to draw from. A small subset of natural forces happen on the experiential side of human scale (magnetism, surface tension, aspects of gravity, friction) ⁽⁴⁾. AI-based generative UIs allow for seamless interpolation through the latent spaces of these styles and configurations.

While wholly speculative magic/sci-fi interfaces remain the upper imaginative bound of what can be done with interfaces, everything is still beholden to human factors – cognitive and motor aspects largely influence what interfaces are practical.

Input Constraints

For visible interfaces, comprehensibility, input resolution, reliability, and latency (critical characteristics of throughput) govern the space of what can be meaningfully selected on a GUI. These get more stringent as they move from abstractions to the real world. Mouse/trackpad input, which has not drastically shifted since the 1960s, accomplishes all three of these well and provides an expansive canvas for GUIs.

Other forms of input impose different restrictions. Phone-based touch controls require a smaller space with larger hit targets but are suited to swipe gestures. Voice has many degrees of freedom but lacks continuous, fine-grained input. Hand/motor tracking may offer a wide range of continuous freedom at the expense of input noise and latency.

Conversely, input systems limited to cardinal directions, like TV browsing UI with a remote, are reliable but necessitate coarse, focus-based navigation patterns.

The same directional input characteristics exist for EMG gestures (albeit with other tradeoffs), so there exists an ideal shared pattern between both approaches.

The same, principled grid pattern designed for remotes/hand gestures would feel cumbersome for voice or cursor-based approaches. If we know the characteristics and constraints of the input we can rule out a swath of UI patterns that aren’t optimal.

Environmental Constraints

Interfaces and their representation have a strong relationship with the environmental context (shifting and composite input modalities, form factor, ambient conditions, etc.) they exist in. Changing your route mid-run on a smartwatch is very different from adjusting a weekly schedule on your phone.

Modern interfaces increasingly aim to predict user intent and outcome through signals – usage history, mental state, and world cause and effect.

Admittedly, real-time/generative responses to the environment introduce the most possible variation to a UI’s appearance ⁽⁵⁾. Regardless of how an interface evolves, though, it’s still tethered to the enduring heuristics of human factors ⁽⁶⁾. Probabilistic interfaces yield better and worse patterns for the situation, assuming the interface has a salient mapping and response to user/world state.

Everything should pass through the “hourglass” of our control and cognition.

Bringing it together

Extrapolating from here, it’s not hard to imagine formulas and generative approaches for whole classes of screen-based interfaces. Perhaps this is solvable.

To phrase this as a hypothesis:

“There are only so many practical arrangements for an interface given how we can interact with it, the environment, and its objective.”

Or propositionally:

“For every interface, there exists a set of viable options (equilibrium) between available human input, environment, and required output.”

$$\forall I,\ \exists E : E = I \cap H \cap O$$

The space of this expands as new types of input/output become available, and the complexity explodes moving from individual UI elements to whole patterns, experiences, environments, and culture. These are the spaces where taste, human intuition and scientific exploration flourish 🌱🪲.

Footnotes