Mixed reality is the next shiny thing for marketers, but some remain cautious

Fifteen years ago, few people could have predicted the change smartphones would bring to advertising — going from novelty to necessity and hastening the decline of PCs and linear TV while sparking the rise of social media and streaming entertainment.

Now, as smartphone sales plateau, a new crop of augmented and mixed-reality (AR and MR) devices could once again revolutionize how we experience technology and reach engaged audiences. Some top marketers, however, are not quite convinced — at least, not yet.

Apple announced last week that it will begin selling the Vision Pro headset at the start of February, and the device is already predicted to sell out. During the 2024 Consumer Electronics Showcase (CES), Sony announced it’s building an MR headset slated to launch later this year — as is, reportedly, Samsung and Google’s.

Early adopters can already choose from AR and MR devices from Meta and buzzy upstarts like XREAL, Rabbit, and Humane. Speaking of the Meta Ray-Ban smart glasses, Niantic CEO John Hanke told the Financial Times, “I think it surprised a lot of people in terms of how well that was received. I think this version is the right device at the right time.”

These devices enable “ambient” or “spatial” computing, whereby physical and digital worlds converge and our surroundings become our screens. AR glasses, for example, overlay digital content on the real world, and MR devices let users interact with content as if it were part of their surroundings.

For all the promise of AR and MR, marketers who spoke to The Current remained skeptical. “For me, it’s very much ‘wait and see.’ I’m not sold that it is quite where it needs to be for consumers to buy into it,” Josh Campo, CEO at Razorfish, told The Current during CES 2024. “We need to see some kind of adoption rates before we can say that this is a channel we should be doubling down on.”

From curiosities to critical strategies

Lingering economic uncertainty may play a role in marketers’ hesitancy toward MR, unlike the excitement that the industry bestowed upon the metaverse two years ago.

“Right now, marketers are all playing it very safe because of what’s going on in the world,” Maggie Malek, president of North America at Crispin Porter + Bogusky, tells The Current. “If this stuff starts to ramp up more, we [will have] more conversations with clients about advertising in these spaces [and] partnering with these organizations.”

But just as mobile, social media, and location-based ads went from mere curiosities to critical strategies in marketers’ toolkits, the implications of this new crop of gadgets could be profound. Marketers can now tap in to MR experiences with haptic feedback and create hyper-personalized, hyper-local contextual ads.

“The post-smartphone era of advertising, where our surroundings effectively become interactive screens, could lead to a transformative shift in how ads are created, distributed, and interacted with,” says Ellie Bamford, chief strategy officer at Wunderman Thompson North America.

“One significant opportunity lies in transforming the pure ‘spectator’ role of the ad viewer into highly engaging and interactive ad experiences,” adds Jaime Gonzalo, vice president of Huawei Mobile Services Europe. “MR allows for storytelling and brand interaction in three-dimensional spaces, offering a more compelling narrative than traditional ‘flat’ ads.”

As tech firms big and small race to make these next-gen devices appealing and affordable to the masses, we could soon witness advertising experiences like walking through a shopping mall and seeing personalized ads or having a virtual shopping assistant in our living room.

“With advancements in location-based services, ads could become even more hyper-local,” says Bamford. “Walking past a store in the real or digital world could trigger personalized discounts or offers directly to your wearable device, depending on your shopping history or preferences.”

Less screen time, more quality experiences

At CES this year, the flurry of MR product launches and commitment from big tech names like Sony has made some technologists more optimistic than marketers about the direction of the space.

“It’s further evidence that the entire industry is moving in the direction of spatial computing, giving resources to creators to create things in this virtual space,” Michael Miraflor, chief brand officer at venture capital firm Hannah Grey, told The Current during CES 2024.

“Developers are [...] eventually going to want to create things that people want, and people want to be entertained,” continued Miraflor. “They want to be productive. And they want to experience things that they couldn’t experience on devices before.”

Amid this uncertainty, some companies are already preparing for the future by tapping in to technology that’s already here and working. Walmart and Qualcomm are reimagining the retail experience with smart shopping carts, video rails on shelves, and biometrics at checkout. This could further boost the retail media segment of the industry, projected to surpass TV spend by 2028, by providing yet more opportunities to engage with shoppers.

Sensory advertising, like Starbucks’ activation in London last year that featured sound effects of ice shaking and scent dispensers, could also be amplified by the rise of wearable tech — from smartwatches to fitness rings — which is projected to see double-digit growth for the next five years.

“The focus should be on creating meaningful interactions that respect the user’s desire for less screen time and more quality experiences,” says Gonzalo. “This could involve more contextual and situational advertising that feels like a natural part of the user’s environment, rather than an interruption.”

How Augmented Reality and Virtual Reality Are Changing Things for Marketers

Every so often, new and shiny marketing objects come along and become the subject of a great deal of hype about how they will change the world. The phenomenon is so typical, that Gartner developed a methodology to represent where each new promising technology is in the "hype cycle."

Of course, some new technologies don't deliver, and yet others prove to be transformational. Usually, however, changing the world takes longer than initially thought.

Marketers are now talking (again) about how augmented reality (AR) and virtual reality (VR) will change marketing, so it's timely to take a step back and put a practical spin on what is likely to happen and what is simply hype.

Augmented Reality

AR is the superimposition of digital data and "objects" into a real-time camera view of a live environment. This technology has been around for a long time, but has only recently begun to be significantly applied to marketing.

There are two basic versions of this technology: Overlay AR and 3D AR.

In the Overlay AR environment, the digital data or objects are not linked to the scale or specific layout of the physical world. The digital object is simply triggered by some event or data (such as a GPS coordinate.) A good example is the Pokémon Go app from Niantic Inc.; users can search for imaginary Pokémon characters, and when they "find" them the character appears superimposed on the video stream from their smartphone camera.

• The digital objects can be images, or they can also be other data (such as video; audio; numbers, such as outside temperatures and coordinates; or text data, such as names of buildings.)
• Often, an object is placed using GPS coordinates (as in the case of Pokémon Go) or it's triggered via some sort of visual cue, such as a QR code or a special image placed on a card.

3D AR is an innovation of Kaon Interactive (my company). A digital 3D product model is placed into the camera view of the physical space, with some key characteristics:

• The object is scaled to fit, with the appropriate dimensions, relative to the physical space around it (e.g., a small object that fits on a table, or a large floor-standing object, both of which have the correct dimensions relative to the table or the floor and walls).
• The object is placed in a position without the use of the visual cues or special cards, in a way that allows the object to "remain" in the same place even when the camera (the phone or tablet) is moved away and then brought back to where the object was placed. Think of it as object permanence, in a digital sense.
• The object is interactive: The user can open doors, remove parts, spin, turn etc. while it is being viewed as if in the physical environment.

Virtual Reality

With VR, users put on a headset or goggles, and they are "placed" into a digital/virtual representation of a physical environment so that they can look and explore it as they would in the real world, as if they were in that environment. When they move their head, they see the space as if they had looked in that direction in the real world. This effect is known as "immersion."

VR usually uses a headset either with a built-in vision system and dedicated computer or a headpiece that simply encapsulates the user's smartphone to use as the display and computer system. Examples include Google Cardboard, which allows users to use their own smartphones in a headset experience, and Oculus Rift, which is a high-definition headset that connects to an external computer.

In many VR games, in which the user can "drive" a car or "fire" a weapon, the application reacts appropriately to that user's action in an interactive manner, giving the player a sense of presence and action.

360 Videos

Another application type, the 360-degree video, is often confused with AR or VR. However, 360 video is a video of an actual (real life) scene that is taken with multiple cameras/lenses pointed in every direction at the same time. The effect is to give the viewer the option to "turn" in any direction and see the action unfold from that perspective, in real-time. (Here is an example.)

The user can pan across the video using the mouse (or touching the screen on a tablet or phone) while the action continues. There is usually no interactivity in this kind of video, other than the user's changing the viewing perspective. The storyline follows the same linear path, but can be experienced differently, depending on which direction the viewer chooses to view at any point in time.

These 360 video experiences can be viewed within an immersive headset (just like VR), on a screen (e.g., on YouTube or Facebook), or in a specially designed room with large screens all around.

Marketing Applications

These new platforms allow for innovative ways to engage customers in an experiential manner. As with all new solutions, the key to being successful is to marry the content of the experience with the appropriate technical approach.

Showing something in VR for the sake of the VR experience has no inherent marketing value to the customer, other than the "cool factor." (In this example, consumer advertisers have been experimenting with overlay AR apps for alcoholic beverages as a way to increase loyalty and engagement with customers. This approach has the "cool factor," but the videos aren't inherently tied to the code on the bottle and could just as easily have been viewed without the AR trigger.) After the cool-factor effect wears off, however, the company will likely not continue to achieve the intended marketing objective.

When it makes sense to overlay marketing information on a real-world scene (say, for example, showing a unique internal product feature that can't be seen from the outside), then there is a memorable and meaningful association of the technology to the marketing message. Customers appreciate these kinds of experiences because they have both the emotional connection and the benefit of learning something relevant and useful.

For complex products and solutions, showing something in a digital immersive environment can be highly effective because the experience is difficult to deliver with real-world products or environments, and the customer is using the immersive platform to learn about the relevant value of the product or solution.

When deciding on how to use VR and AR in the marketing mix, think about what is a practical application given the customer's environment, and how the three key elements of engagement (sensory, intellectual, and emotional connections) will be used to deliver an effective marketing experience.

A couple of examples can illustrate the innovative impact of AR and VR in marketing:

1. 3D AR showing how a new product will fit into a customer's environment. For a consumer application, a good example is Wayfair, which has a 3D AR app that shows its furniture in the customer's home. For a B2B application, applications can show how large industrial equipment can fit into a manufacturing assembly line, or how lab equipment can be placed effectively into an automated clinical diagnostics lab.
2. VR for complex B2B solutions. A recent example is a VR experience in which an IT manager can explore a virtual data center to see how the software-defined network (SDN) digital data reconfigures physical data center hardware in real-time. This is impossible to show in a real-world data center (because the data is not visible, and the configuration of the equipment happens inside the servers and switches.)
   
   These complex hardware and software products can be displayed in an immersive environment that provides the user with the sense that they are in the real data center, but uncovers how the software works, providing a clearer understanding.

Immersive AR and VR applications have the potential to change the connection between customers and companies because of the opportunity to deliver the most engaging, personalized, and useful experiences.

Judicious selection of the appropriate tools and venues will provide innovative marketers with effective and valuable solutions.

Why Virtual Reality Will Change Design Forever

MARK WILSON 04.11.16 7:00 AM

My dining room may never be the same. The Ikea table and vintage fiberglass chairs have been pushed aside. The perimeter is traced with dangling cords, precariously connecting a constellation of hastily tape-mounted laser trackers. A glowing PC is the room’s new focal point. My toddler is unsettled bythe alien's neon eyes.

But it’s all worth it, to carve out a patch of open land in my cramped condo where I can put on the HTC Vive, grab two motion controllers, and experience the most advanced incarnation of virtual reality on the market today. Six years after the iPad came out, we’re still debating whether or not you can actually make things with it. But I hand the app Tilt Brush—imagine Microsoft Paint in 3-D space—to my wife for the first time. And within minutes, she’s crafted a flowing, rainbow dress in midair.

RingGrub by Issac CohenIsaac Cohen/cabbibo.is

I know, this sounds totally insufferable. But you—casual, budding, or established designer—should do whatever you have to do to try an HTC Vive, too. Because it’s not actually flying around in space or watching whales at the bottom of the ocean that makes VR incredible. It’s that in this second reality, digital experiences don't just seem physical—they feel physical. The fourth wall of your monitor is gone.

Inside this space, creating content is as exciting as consuming it. And design will never be the same because of that. Here's why.

Tilt BrushGoogle

IT'S TANGIBLE

The sensation of creating in virtual reality is nothing short of godlike the first time you try it. Of course, it doesn't hurt that you can literally draw in glowing light, burning fire, or shimmering stars. But it’s gutturally remarkable because you can walk around the room, viewing these creations from any angle—even crouching down or moving your nose just inches away—and your mind accepts them as real. Or, real enough.

You’ll never actually forget that there’s a 1.2 lb pair of goggles on your head, nor will you be able to ignore the omnipresent pixels that rewind graphics fidelity to about a decade into the past. Yet as I step through different spaces, from a N.Y.C. subway car to an Ikea kitchen to a warlock’s den (look, VR is early, and my choices are limited, okay?), my gut believes that I can bump into the pole, or the waist-high tables in front of me.

Ikea

And so I find myself sometimes inching my steps or sucking in my tummy to navigate a virtual space. At one point, I force myself to walk into the sink of my Ikea kitchen, just to remind myself that I can. My heart dropped for a moment, but I lived.

Your mind knows virtual reality is fake, but your limbic system isn’t so sure. And when you combine that phenomenon with how good creation feels in virtual reality, design gets interesting, fast.

IT'S VISCERAL

The sensation of drawing in all dimensions is instantly addictive. It’s kind of like 3-D printing with your mind. It takes just a thought in your head and a swirl of your arm, and a sculpture manifests from nothing.

A filmmaker named Isaac Cohen recently shared his work with me. He said that since he started working in VR, he’d become interested in strange, experimental spaces, rather than narratives. He began experimenting, creating shimmering tubes and iridescent yurt-like installations. It wasn’t until he talked to an architect he knew that he learned he’d stumbled onto the rich tradition of "liminal spaces," which perfectly described the transitionary, ritualistic, and otherworldly rooms he was intuitively designing.

meta cloth 1 by Issac CohenIsaac Cohen/cabbibo.is

Cohen's architectural experiments are just one example of how VR makes a broader range of creative mediums attainable to users. He tells me how his friends have consulted a fencing instructor to master Fruit Ninja slicing techniques, and painters to improve their mock-ups of industrial design. It makes sense though, doesn't it? Lower the barrier on building a wall, and anyone can be an architect. Give someone the ability to paint in light, and they can be Picasso.

We've seen this effect on a smaller scale over the past decade; for example, desktop and mobile video editing software allows anyone to edit and upload their work to YouTube. But VR puts this phenomenon in real space. Skills that would normally take years to develop, and tools that would only be accessible to professionals, are immediately available and completely immersive. Anyone can be a Renaissance man.

IT BUILDS EMPATHETIC EXPERIENCES

Designers spend a lot of time and money to understand what it's like to be somewhere, in someone else's shoes. But the thing about VR is, not only can you use it as an empathy engine to experience life as someone else—you could theoretically design in someone else's shoes, too.

A recent VR video by Pencils of Promise—a nonprofit that raises money for international schools—placed donors right into the desk of an African school. In first person, it’s a surprisingly powerful experience. You can almost feel the heat off the mud brick walls, and what it must be like to sit through class as a bored grade schooler in that village.

Pencils of Promise

Now imagine if you could pause that moment and change something in the scene to improve the experience: You could add a window to see how it increased light. You could craft a taller chair that allowed the back row students to see over the kids in the front of the room. Or maybe, if you were really ambitious, you could simulate running an iPhone loaded with your startup’s latest app in that space.

When designers can travel anywhere and be anyone, design changes forever. No longer will creatives be forced to mock up some new invention in a Bay Area cubicle before taking 10 hours to build it in a 3-D printer. They'll sit on the ground in Africa, or the bottom of the ocean, or the N.Y.C. subway car, and use a magical paintbrush to paint the reality they can otherwise only imagine.

So despite all the cords, I’m going to take a few days before moving my dining room table back, feng shui be damned.