The term “artificial intelligence” is likely to have stiff competition this year when it comes to being the buzziest phrase in tech. That’s because Apple is expected to launch its Vision Pro mixed reality headset, and simultaneously, Apple hopes, bring the words “spatial computing” into the mainstream.

Here’s what you need to know about this new frontier in tech and how businesses may use it.

WHAT IS SPATIAL COMPUTING?

The concept of “spatial computing” is widely attributed to researcher Simon Greenwold, who coined the term in a 2003 paper. Greenwold defines spatial computing as the “human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces.”

A simplified way of saying this is that spatial computing melds a computer system’s user interface with the real world. Using a spatial computer—usually in the form of goggles—the wearer can still see the real world around them, but they can also see other elements overlayed on top of this real-world environment and interact with those elements in the physical space using their bodies, typically hands.

HOW IS SPATIAL COMPUTING DIFFERENT THAN AR AND VR?

In VR, you are completely immersed in a virtual world and can see none of the real world around you. A VR device doesn’t know a couch is in front of you and doesn’t care. VR is ideal for things like video games. Sony has a dedicated VR gaming device called the PS VR, which is a great example of a consumer virtual reality device.

Augmented reality (AR), on the other hand, displays virtual elements on top of the real world. Whether you know it or not, you almost certainly already own an AR device: your smartphone. Tons of apps take advantage of AR technology to let you see data layered atop the real world through your smartphone’s screen and camera. These apps include navigation apps that display virtual arrows on your screen, so you know where to make the next left, or gaming apps like Pokémon Go, which let you hunt and find digital Pokémon in the real world. AR apps know more about where you are in relation to the real world than VR devices do because they utilize other tech in your smartphone like GPS and rear cameras.

Still, with AR, you are forced to interact with those digital overlays via your smartphone’s screen. Though you can see the virtual Pokémon on your screen, if you reach out to where it looks to be situated in the real world, you wouldn’t be able to manipulate it in the physical space before you.

And that’s where AR and spatial computing really differ. In spatial computing, you can interact with those virtual objects by simply using your hands in the physical space in front of you. Thus the “space” in “spatial computing” refers to the ability to act with virtual objects as if they were actually physically present in the real world. What’s really cool about advanced spatial computers is that the virtual elements displayed can also drape virtual shadows across the contours of real-world objects to give the virtual ones a sense of depth and physical presence.

A great fictional example of spatial computing is that scene in the first Iron Man movie where Tony Stark is using his bare hands to manipulate the design of a virtual Iron Man suit before him. Of course, spatial computing that advanced is still a ways off as all our current spatial computers require goggles.

WHAT ARE SOME CURRENT EXAMPLES OF SPATIAL COMPUTERS?

The best example of a spatial computer is the Apple Vision Pro, which Apple previewed last June and indicated it will release in the early part of this year. The Vision Pro is a headset that you wear that projects virtual elements into the world around you. You can interact with those elements by just using your hands, eyes, or voice—no tapping on a screen or mouse needed.

But Apple won’t be able to lay claim to having released the world’s first consumer spatial computer. Plenty of other companies have spatial computers on the market, including Microsoft with its HoloLens 2 and Meta with its Quest Pro.

HOW DOES SPATIAL COMPUTING WORK?

Spatial computers can do so much more than augmented reality devices can because they are packed with more sensors, cameras, and other input technology that can detect and read the world around you in real time. For example, Apple’s Vision Pro has 12 cameras and five sensors that help the device know everything from the level of light in a physical space to where objects are in relation to each other, to where your eyes are looking, and how your hands move.

The sheer number of cameras, sensors, and other advanced tech contained in the Vision Pro is one of the main reasons the device costs an eye-watering $3,499. Less advanced spatial computers like the Meta Quest Pro cost $999.

HOW CAN SPATIAL COMPUTING BE USED IN THE WORKPLACE?

Consumer uses for spatial computing include everything from video gaming to media watching. A spatial computer like the Apple Vision Pro can make it feel like you have an IMAX-size screen in your living room. But what about businesses? The use cases there are a bit more niche, but no less promising.

One of the most obvious examples of how a spatial computer can benefit office workers is by giving them an unlimited number of virtual displays to view all their open documents side by side. Goodbye, tabs! A single spatial computer headset can eliminate the need for multiple physical displays, so if you are working with multiple spreadsheets, PowerPoints, or photos, you can project a dozen of these kinds of documents in the space in front of you, move them around, and adjust them to appear as large or as small as you like.

Design and engineering are fields particularly ripe to benefit from spatial computing. Designers and engineers already use advanced 3D modeling programs to conceptualize their products. Now, with spatial computers, those 3D models can move from the flat 2D monitor to the real world around them. For example, an automotive designer can walk around the virtual version of her concept car right in her own garage, seeing if the car’s slopes and angles look as appealing in real life as they do in her CAD drawings. Or an action figure designer can test out different-size prototypes of the hero to see how the figure works with existing vehicles in the toy line without having to actually manufacture any.

When it comes to setting up an office or workspace, a spatial computer can save a company from having to manually move objects like desks, couches, chairs, or machinery unnecessarily, thus saving time and expense. If the business can render digital twins of all the components that they want in their workspace, they can simply arrange and rearrange the virtual twins in the physical space first, and with the spatial computer goggles on their faces, they can walk around the empty space to make sure that everything fits right and people can still move around easily. The same advantages apply to retail stores that want to test out new merchandising layouts.

Spatial computing can also help businesses train new employees. A car repair shop could use a spatial computer to guide a new hires through replacing their first carburetor. It could display arrows pointing to which bolts the new worker needs to undo in what order, leaving the manager to attend to other things.

And let’s not forget the communication benefits of spatial computing. The technology can bring video conferencing into the 21st century by freeing long-distance communication from a flat computer screen. Using spatial computing, the people you are video conferencing with could appear as if they are sitting across the desk from you, or huddled around the whiteboard with you, leading to a more natural, interactive experience than traditional video calls provide.

In short, spatial computing does truly open up a new frontier in the computing space for consumers and businesses. With the upcoming launch of the Apple Vision Pro, expect to hear the term a lot more in 2024.