The Day That Everything Changed

ARKit and ARCore bring immersive computing at scale. What we will build?

June 5, 2017 was the day that everything changed. During Apple’s 2017 World Wide Developer Conference, the company announced the beta availability of ARKit, the native software layer for blending digital objects with the real world. In the four short months since that announcement, ARKit has taken the world by storm, unleashing a series of developments that will not only put augmented reality at the forefront of mobile computing, but also provide new momentum for virtual reality. And just in time.

AR’s Long and Winding Road

Augmented Reality had been wandering in the wilderness for some years. The explosion of the smartphone market led to a crop of startups circa 2010 that brought the camera and location services together with innovative applications, such as Metaio and Layar. These types of ventures enjoyed mixed success, at best, getting acquired or pivoting, as the market was slow to materialize.

One notable exception is Vuforia, a supplier of AR middleware that spun out of Qualcomm and was purchased by CAD software giant PTC. Vuforia has continued to find purchase via a combination of enterprise and consumer applications, and powers tens of thousands of AR experiences today. But for years, even Vuforia’s AR was considered a “gimmick” for many applications, not an essential enabling layer of what many of us believe is the next step change in computing, the immersive interface.

Then along came VR.

Back In the Limelight

View at Medium.com

#AR: Thanks, VR! Wouldn't be here without you.#VR: 🙃

— Tony Parisi (@auradeluxe) July 30, 2017

Looking back on the stratospheric rise of virtual reality since the Oculus Kickstarter five years ago, I am still amazed. I am obviously jaded about this space, having worked in it for over twenty years, so I was very skeptical about the possibilities when I first tried the DK1. It wasn’t just the low resolution, nausea-inducing tracking and insufferable form factor; those should have been enough to kill this thing in its infancy. No, I was more skeptical about the market, because I had been down this road before. I couldn’t imagine that consumers were ready for an immersive computing experience, because I had seen too many failures, both personal and industry wide, that were clear indications that the world was not ready for 3D.

Obviously, I was wrong about Oculus. Enough things had changed in recent years, apparently, that folks could see the potential in a fully immersive computing experience. Not just the tech industry, but consumers and customers. And so, game on: here we are, five years into the immersive computing revolution, thanks in large part to the Oculus Rift.

With the resurgence of consumer VR came a parallel renaissance in augmented reality. AR industry players wisely took advantage of the spotlight VR cast on immersive technology, and renewed their marketing efforts, riding on its coattails.

This doesn’t mean that AR was standing still that whole time. Pokémon GO and Snapchat filters shipped as mass-market consumer AR phenomena, followed by Facebook’s camera-based AR premiered at F8 this year. Also, Microsoft Hololens and Google Tango have been pushing the envelope on industrial AR hardware for several years. But I think it’s fair to say that these projects have enjoyed additional consumer awareness due to massive buzz that was building thanks to VR.

And right as AR stepped back into the limelight, ready for its closeup, along came ARKit.

Immersion at Scale

On the heels of ARKit’s release, Google announced ARCore, an Android API that brings a subset of Tango’s AR technology to consumer-grade android phones, notably Samsung S8 and Google’s Pixel line. ARCore is similar in features to ARKit, offering single-plane surface recognition, positional tracking — so you can blend true 3D graphics, not just sprites overlaid on your camera image — and light estimation.

ARKit and ARCore leave out some of AR’s more promising features, some of which require additional hardware support, such as environment reconstruction (aka SLAM). But it’s reasonable to think that, over time, those core technologies will migrate into newer generations of mobile devices. In the interim, smart software libraries like Vuforia can fill the gap with a combination of on-device APIs and cloud-based services (not to mention much-needed provide cross-platform capability between the two operating system APIs).

While the feature sets for ARKit and ARCore don’t run the gamut of what many of us would like to see for “true” AR, this is a significant development nonetheless. Because with these two systems, we now have a baseline AR capability that spans the two most popular mobile ecosystems, and reaches nearly half a billion devices.

And Apple isn’t standing still. Today, the world changed again with the company’s press event about the upcoming iPhones. iOS 11 ships on September 19th, and with it, the ability to deliver AR to between 300 and 400 million phones. The iPhone 8, coming out on September 22nd, has even tighter integration among the components that run AR. And the iPhone X, previewed today, comes with Face ID, based on a TrueDepth camera system that has a depth camera, IR camera, flood illuminator, dot projector, and an on-device neural network on a chip (the A11 bionic chip… what?) — the better to recognize you and only you, securely. The same face tracking tech is also used to drive true 3D AR filters as well as animojis— cartoon characters that speak for you in your instant messages.

It’s safe to say that Apple appears to be doubling down on its AR bet.

Now that the big guys have stepped into AR with full force, the world has taken notice. We’re seeing a huge wave of interest in AR from game developers, storytellers, brand creatives, and enterprise application developers. It’s fair to say that we could be approaching a tipping point for immersive development that we hadn’t seen with VR, because now AR promises scale, based on the phone in your pocket, with no need to strap anything funky on your head.

VR Will Have Its Day (Again)

This reinvigorated interest in AR comes at a time when VR is sputtering a bit. Many of us knew that mass-market adoption of VR was going to take a while. Unity’s CEO (and my boss) John Riccitiello spelled this out in a keynote address at the Vision Summit in 2016, a talk which has come to be known as his “gap of disappointment” speech. The basic idea was that, while we’re bullish on the long-term growth of XR, we know that things are going to take longer than a lot of folks expect.

John also did the keynote at this year’s VRLA conference, and his talk there featured a slightly more upbeat, updated version of this concept. John laid out what he thinks it’s going to take, specifically, to get there with mass adoption of XR. Have a look if you haven’t seen it.

The biggest takeaway from this talk is that we need two critical things to hit mass scale: 1) a total cost of ownership of under 1,000 dollars, and 2) a couple of hundred million units shipped.

VR is nowhere near that yet. While costs are quickly dropping, we are generously at 10 million units (not counting Google Cardboard; but that’s a debate for another day).

ARKit and ARCore, on the other hand, get us to that mass scale in one stroke.

Now, John wasn’t talking about AR in that particular keynote. He was focused on VR. But these two technologies occupy a spectrum of immersion. At the highest level, VR, AR and headset-based mixed reality are all about immersive 3D graphics. Yes, there are some points of divergence. But in general, it’s real time 3D objects, environments and characters viewed in an immersive 360 degree environment. The skills you need to build for one translate well to the other, and consumers will become more and more ready, even expectant, for interactive 3D content regardless of its delivery medium. And of course, you can use Unity to build for all of them. Mwahah…

The recent big developments in AR threaten to steal VR’s thunder. We are already seeing a bit of an exodus by developers attracted by the lure of scale and the potential for monetization that it brings. Despite VR offering a more complete and compelling trip, the user base for AR is two orders of magnitude higher. Why invest time and energy into creating something that can be seen by 10 million people, when you can put that same energy (actually, quite a bit less in most cases… AR content tends to be much less complex) into making something for an audience of half a billion? It’s hard to argue with those numbers.

But I am guessing this is simply a stage in the ongoing development of immersive computing. In the balance, I think that it’s going to be a good thing for AR to get the attention for the next little while. VR needs more time to get the pricing and form factors right. But there’s no replacement for the sheer joy of completely enveloping yourself — head, hands and body — into an experience that takes you to another place. VR enables magical realms that AR can’t, by offering complete escape. The pendulum may swing to AR for a while, but there are so many great use cases for VR that I believe it will have its day again, and the fresh energy that AR brings to the world of immersive computing should give it enough lift to keep VR rolling too.

Blurred Lines

The Art of VR brought together VR, AR, MR and traditional art into a mashup of media that gave glimpses into the future of the visual and digital arts.

Surreal kickoff to a surreal event: I opened The Art of VR from the auction podium at Sotheby’s New York!

[Better late than never dept.: sorry been on the road for a month.]

I go to many XR events these days, so you can imagine it takes a lot to keep my interest. Well, The VR Society’s The Art of VR was something completely different! It was hosted at the historic Sotheby’s Auction House on New York’s upper east side. Not a place you would associate with the typical tech expo; but this wasn’t a typical tech expo. The Art of VR, as the name implies, was focused on art: digital art, interactive art, and novel interplays between traditional art media and XR.

This last bit is what really excited me about the show. There were several Made with Unity pieces that blurred the lines between traditional, physical art and digital works in XR, by bringing the physical into the virtual, or augmenting the physical with the virtual. I even worked on one of the featured pieces by creating an experience using Unity and Apple’s new ARKit — a lovely change of pace from my nine-to-five of slinging strategy PowerPoints and prancing on stage giving keynotes! Here are the highlights.

Simulcra by Isaac Cohen

First up, Unity’s new Artist in Residence, the inestimable Isaac Cohen brought his usual insane flair to the proceedings. Two days prior to the show’s start, Isaac, better known by his nom de code Cabbibo, got to the venue and did a photogrammetry scan of his exhibition space. He then converted the couch into an elastic/blobby/furry/primary-colored piece of psychedelia that you can tug, pull and play with.

“Simulacra” by Isaac Cohen (aka Cabbibo)

Or, because the virtual space is a one-to-one recreation of the physical space, you can actually plop down on the couch! Here is one particularly satisfied customer:

Cabbibo whu?

Zenka: Art for the Galactic Age

Jenny Carden, aka Zenka, a sculptor and futurist who specializes in “Art for the Galactic Age,” has been exploring the intersection of the physical and virtual in her work for some time now. Her ceramic sculptures heavily reference XR, with human heads sporting outlandish HMD eyewear. Her paintings are launch points for augmented reality experiences that combine the 2D wall art with live animated CG content via a mobile app. Jenny has been pioneering this art form and using software tools like Vuforia, Aurasma, Blender and Unity.

Jane Lafarge Hamill’s Wind in the Woods

Abstract painter Jane Lafarge Hamill premiered her first foray into VR, Wind in the Woods. This is an interactive piece for the Vive using Unity. In this work, you begin at a marked spot on the floor, facing one of her paintings, and then pop into the Vive to see a virtual version of the painting in VR located in the same spot as in real space. Start walking, and — in a truly wondrous moment — you actually enter the painting, a virtual recreation of the made with Unity. You can move around the piece with the Vive’s room scale tracking, looking in and around 3D objects extracted (extruded?) from the original painting’s abstract shapes.

Jane Lafarge Hamill’s Wind in the Woods

Floating World by Marina Berlin

Painter turned sculptor Marina Berlin is my better half, and (true story) the person who long ago got me into virtual reality. For the last four years, Marina has been working in industrial wire mesh, aka chicken wire, creating realistic sculptures of people, animals and fantastical beings. Her piece Floating World depicts human figures suspended in mid-air, wearing head mounted displays, immersed in a virtual experience. There is a companion piece for Gear VR that is a virtual visit to her studio, where you can watch Marina build one of the figures in a time-lapse video. For The Art of VR, I added another layer: I develop an augmented reality experience to complement the real-world art. Wire-frame rendered birds flock about the human figures, so that you can see the virtual creatures our wire people are experiencing in VR. I built it in Unity using our new ARKit integration. It was lovely to collaborate with Marina, and a breath of fresh air to do some hands-on development!

Floating World, by Marina Berlin

I’m super excited by what happened at this show, proud to have been a part of it, and hopeful that we will see more of this type of art that blurs the lines between the physical and digital. Check out more of these innovators’ work on their websites!

Face the Future

Computing in an Augmented World

[Transcript from my keynote talk at AWE 2017.]

We live in a 3D world. People move, think and experience in three dimensions.

But for decades, human-computer interface has been… less than that.

As computing continues to get woven deeper into our daily lives, in many ways, interface hasn’t caught up. We still operate our computing devices — or perhaps more accurately, they operate us — at some remove from how we work within the real world.

From punch cards to command lines to GUIs to touch screens, the computer industry has seen a steady progression in usability; but there remain levels of abstraction compared to how we interact with our surroundings and each other on the physical plane.

Today, we peer into an endless sea of information via tiny portholes… flip through pages and tabs linearly while trying maintain a mental map of things and their connections of much higher than one dimension … and remember and share moments from the world around us by capturing only the photons in front of us.

Tomorrow, information will be displayed where and when we need it, its essence and connections laid bare to see… and whole experiences will be captured and shared. We are about to escape the tyranny of the rectangle… and information itself will achieve six degrees of freedom, in a virtual sensorium that maps the web onto the real world and brings the real world into the web.

Over the past several decades, every time people made computers work more like we do — every time we removed a layer of abstraction between us and them — computers became more broadly accessible, useful, and valuable to us.

— Clay Bavor, VP of Virtual and Augmented Reality, Google

With the advent of immersive technology we are approaching the next step change in human-computer interaction. As HCI goes 3D, some truly staggering things are going to happen. Doubtless this transformation to an immersive computing landscape is going to make digital technology, in Clay Bavor’s words, “more broadly accessible, useful, and valuable to us.”

We’re here today because we all know that immersive computing represents the next platform. But most of us can’t really imagine the ways it is going to disrupt our daily lives, any more than back in the day we could have imagined the Internet disrupting brick and mortar retail, or smart phones upending the transportation and hospitality industries.

We’ll explore some of the possibilities momentarily. But first, let’s ask ourselves why now? Real time 3D is nothing new, and we’ve been tinkering with virtual and augmented reality hardware for decades. Why do we think now is the time to collectively invest so much energy as an industry?

3D graphics is nearly as old as the computer itself, tracing its roots back to the 1960s. It has been used in applications spanning engineering, education, training, architecture, finance, sales and marketing, as well as gaming and entertainment. Even the first immersive VR systems were demonstrated over fifty years ago.

Historically, 3D applications relied on expensive high-end computer systems. But that has changed in the last decade. 3D processing hardware is now shipped in every computer and mobile device, with the consumer smartphone of today possessing more graphics power than the professional workstation of a decade ago.

Look around: a lot of our media is already 3D — though until recently it’s been presented on flat screens. Animated films are created from computer-generated 3D images. Video games, whether running on dedicated consoles or mobile phones, are typically rendered in 3D. And even the news has gone 3D: the sight of a CNN analyst meandering through a virtual set, comically awkward a few years ago, has become an accepted part of the broadcast milieu as cable channels vie for increasing attention in a 24-hour news cycle.

But until very recently, 3D was still lurking at the margins. Other than professional design software, the only widespread use of 3D has been in video games.

In my opinion, that’s because historically, 3D has been a luxury for most applications. The benefits of real time interfaces on flat screens were simply never worth it, other than for niches in design, engineering and filmmaking. Gaming is the one exception — the only segment of the computer industry that has been able to solve simultaneously for the high cost of producing killer content, and monetizing it at scale.

Well, all of this changes with XR.

Inexpensive stereo displays, holographic optics, binaural audio, 360 cameras, volumetric capture, and computer vision, combined with the previous innovations in mobility, low-latency networking and location services, are enabling applications even the wildest of us 3D dreamers couldn’t have even dreamt about a decade before.

And with immersive hardware, we have no choice but to render in 3D. The entire graphics pipeline is set up for it. We have a 360 degree space in which to work. Even if we only display textured rectangles in that space, we need to lay them out using 3D coordinates. But of course we can do so much better that that: we can create virtual spaces and display virtual objects. We can organize information the way we want, using spatial memory and spatial reasoning to get the most out of it. And all this content can be blended seamlessly with the real world around us.

Beyond cheap hardware, we also now have millions of content creators savvy in 3D development– thanks to the video games, design and VFX industries– and most importantly, an interactive generation who expects — nay demands — this. Put a print magazine in front of a three-year-old. What’s the first thing she’ll do? Tap it; pinch to zoom. Today’s young kids are tomorrow’s XR natives.

With XR, real time 3D is not a sideshow; it the main attraction.

With XR, there’s nowhere to go but in.

While much of the focus today is on the innovations on the hardware side, software, content and services are where most of the value will be created over time.

I’m sure you’re familiar with my company, Unity. We occupy a unique position in this new industry.

For those that aren’t fully aware of us: Unity is the creator of a flexible and high-performance end-to-end development platform used to create rich interactive 2D, 3D, VR and AR experiences. Our graphics engine and full-featured editor enable the development of beautiful games or apps and easily bring them to multiple platforms: mobile devices, home entertainment systems, personal computers, and embedded systems. We also offer additional solutions and services including Unity Ads, Unity Analytics, Unity Asset Store, Unity Cloud Build, Unity Collaborate, Unity Connect and Unity Certification, all focused on helping our to developers succeed.

By the numbers, here are some amazing stats: 38% of the top 1,000 free mobile games are Made With Unity; Unity games triggered 5B app installs last quarter alone, across 2B unique mobile devices. When it comes to VR and AR, around 58% of developers surveyed are Unity users, and over two thirds of all VR/AR content released through known channels is Made With Unity.

You probably know many of the top Made with Unity VR games like Job Simulator and I Expect You to Die, and popular VR experiences like TiltBrushand new Cinematic fare like Baobab’s ASTEROIDS. And Facebook Spaces, the company’s new VR social application, recently shown at F8 and Unity’s Vision Summit.

Add to this that over 80% of Vuforia AR applications, and 91% of all Hololens mixed reality is made with our software… you can see that Unity has become the Foundation of VR/AR development.

So, how did we get here? Over the last couple of years, we invested heavily in supporting VR and AR hardware platforms, forging deep partnerships with the major HMD providers.

But we couldn’t have even gotten into that position without first having built a solid foundation for our business. Unity was founded one three core principles, to which we remain committed to this day:

•Democratize development — everyone can bring their creations to life, including a new wave of creators in AR and VR — many of whom do not have game development experience

•Solving hard problems — supporting almost every platform on the planet, and continuing to invest in the best graphics and performance

•Make developers successful — making a profit is hard… but we’re proud to be able to say that in 2016, we paid out more to developers than we’ve made ourselves

Understanding Unity’s position in the ecosystem, now let’s look at some of the great XR being made with Unity, in particular augmented and mixed reality that is pushing the envelope on user interface.

For brands: Rewind Studios created Flight Deck, an application that combines HoloLens with 2D screens to create a whole new type of viewing experience for the Red Bull Air Race. In the HoloLens, the user can actually put the race course on a coffee table or likely even the floor — for a real bird’s eye view — and watch in real time as the pilots make their maneuvers. This piece envisions both brand experience and live sports in a whole new way.

For work: ScopeAR “puts KNOWLEDGE WHERE YOU NEED IT”… they created Remote AR, a telepresence collaboration tool that allows remote field technicians to connect to experts via live video, audio, and a powerful annotation toolset that is augmented to ‘lock’ onto the real world with augmented reality.

Scope used Unity to create and manage 3D objects, while being cross platform and supporting many different variations of devices, from phones and tablets to AR glasses to desktops. And taking advantage of Unity’s robust 3D rendering and plugin architecture they built an entire real time audio-video stack along with the capability to remotely synchronize 3D data (meshes from Tango or Hololens for example, or 3D drawings using line renderers, or complicated CAD models).

Nexus Studios took outdoor adventure to a new level, by creating an AR experience for The Gruffalo, the best-selling picture book. Families follow clues on an interactive trail in the English forest, tracking signs of their favourite characters. These clues lead to different Augmented Reality markers — footprint signposts,specific to each one of the characters. When the visitor aims their device at the footprint marker, a short animation of that character is triggered, blending in with its natural surroundings. The Gruffalo experience shows how we can use AR to turn the world into a canvas for art and storytelling.

SwapBots are collectable & customisable toys that are brought to life by a smartphone or tablet using augmented reality. There are hundreds of combinations of the physical toy, as you swap parts, and endless single- and multi-user game play possibilities.

Mekamon are multi-functional, connected battlebots with augmented reality capabilities. Control them with your smartphone and see power ups and explosions using AR. Fight hostile aliens in AR, play arcade games or engage in warfare with other bots.

Finally, I think we’re all excited about the possibilities for education. MyLabprovides augmented reality educational content for Hololens…

And MergeVR’s Merge Cube is a physical cube that, when viewed through a standard smartphone and the MergeVR headset, displays animated, interactive education content. Wild.

Software is Eating the World

— Marc Andreesen

Even if with all this great stuff going on, you’re still skeptical, well I’m here to say it’s time to suspend that disbelief.

It’s fair to think of the last few years of XR as one big, expensive, global proof of concept. We’re still experimenting with hardware and tools, formats and form factors. We’re still learning the right way to do a lot of things, and the best applications of immersive computing. In fact, we’re just getting started. But the pace of development doesn’t seem to be letting up. Massive, ongoing investments are being made in this space — Vuforia, FB, Snap, Google, MS, and startups like Meta, Magic Leap, Improbable.

So this is happening. And more innovation is right around the corner.

A few years back, Marc Andreessen famously said that software is eating the world. At the time, he didn’t intend it to mean the actual world — he meant information, communications, business processes. But the actual world — well, that comes next.

Google Tango is about to hit the mainstream.

Asus and Lenovo are now showing their new Tango-powered phones, with built-in fisheye camera, depth sensor and infrared sensor. Combine this with powerful software from Vuforia and Unity, and we can blend the real and virtual worlds seamlessly.

And mapping the interior — what Google Maps and StreetView did for the outside, SLAM (Simultaneous Localization and Mapping) does for the inside. Facebook made a big deal about this at their F8 conference, talking about SLAM and object recognition. Google has branded their SLAM-based interior mapping software WorldSense, and Vuforia has its Smart Terraintoolkit for environment and object recognition.

We can use technologies like this to find our way around within a store, museum, or other property. And not just physical objects, but we can heat map wifi coverage, airflow within a room, and more…

But not just mapping the world — ingesting it. In just a few seconds, this Tango-enabled phone scanned a room interior in crude form. With several more minutes’ effort, you can get a high quality scan, bring it into Unity and publish a Daydream app based on real physical environments. All from a consumer-grade smartphone.

With the help of some nifty new smartphone hardware, software is now poised to — literally — eat the world.

Of course it’s not only about getting the right tools in the hands of both the consumers and pros; it’s also about infrastructure: a global information system that allows anyone to publish and share content and applications. Also known as the World Wide Web.

At Google I/O, the company demonstrated a development version of Chrome that extends WebVR — the API that connects web browsers to VR hardware — into augmented reality. WebAR brings the phone’s camera and Tango-style sensing capabilities into the browser, which means that for many use cases, you would no longer need to download a specialized app to recognize markers, scan the environment and overlay information: the world becomes your QR code. And you can easily find cool virtual stuff nearby with one click, because someone shared a link in their social feed.

The digital world will soon be enmeshed enough with the physical world that our ‘reality’ will be the transparency mode that we choose.

— Monika Bielsktye, Creative Strategist/Future Prototyper

VIRTUAL REALITY AS POSSIBILITY SPACE
I’d like to invite you to reimagine what content could mean as Virtual Reality is finally becoming a reality —medium.com

This is all the great stuff that is happening right now.

Now let’s play it forward a few years.

Take it as a given that immersive computing becomes the dominant paradigm: virtual worlds, virtual places, virtual objects commingling with physical space, all presented in familiar, intuitive real-world metaphors, accessible with simple commands and discoverable online. Gesture input and voice, of course. Untethered and fully mobile. Lightweight glasses or even contact lenses. These all seem obvious… once we’ve taken the plunge into immersion.

What’s really interesting is what comes after that, when other computing innovations converge with XR. AI for endlessly variant NPCs in large scale worlds and never-ending stories; deep learning to enable a malleable environment that responds to our every whim. Integrate IoT and we can control stuff in the real world — with our very minds, or as close as we can get.

Last year, designer and filmmaker Keiichi Matsuda released a thought-provoking short film called Hyper-Reality, depicting some of the possibilities of an immersive, augmented digital world.

As computing goes immersive, every inch of available virtual real estate will be up for sale for manufacturers and marketers to hawk their wares. The history of advertising is, after all, the history of a rise in production value. And with XR we so much more production value at our disposal: 3D graphics, animation, 3D sound, and haptic inputs, to name a few. This could make our future world a brand-rich candyland with myriad choices…

Or a hot mess — swipe left to reset your life, your credit history, your social status… as these technologies that promise to bring us closer to our world and each other instead trap us in a Super Sad True Love Story of our own making.

Of course it’s too early to tell how this will all shake out. I’m hopeful, if only because we’ve been here before with other disruptive technology.

One of the safeguards against some of these more negative outcomes is democratization of the platform. By enabling the independent creator, we can level the playing field, and not everything has to be paid for by big companies. A 15 year old kid could become a future YouTube star in XR… or develop cancer detection tools… or create the next XR artistic masterpiece.

Though the early returns are incredibly promising, they’re just that: early. We don’t know which devices and form factors will win; which will be the dominant formats or killer apps. But we do know that the world is going immersive.

Investments made now in immersive computing — in learning the tools of the trade like Unity, in understanding immersive design, in adding meaningful layers to the real world, in short, in building interfaces for people — will pay off many times over in the future. Because that future is 3D.

Thank you — and enjoy AWE!

Vision Quest

Got VR conference fatigue? Me too. But Vision Summit is the one event you can’t miss this spring.

It’s mid-April, and I’m already burnt out on VR events. Not to take anything away from SVVR, VRLA and the other great conferences that have already happened this year — I’m just ready for a break. But Unity’s Vision Summit is right around the corner, and it’s going to be really special. Last year’s inaugural event was a huge success, and this one is looking to build on that.

Full disclosure: I work at Unity and I’ve been heavily involved in putting together the program. As our planning for Vision comes down to the wire, I am getting to the point where I can’t contain my excitement, so I thought I would share a few thoughts about what’s in store at the beginning of May.

Insight, Inspiration, Innovation

Unity CEO John Riccitiello at Vision 2016

At the Vision 2016 keynote, our CEO John Riccitiello shared his thoughts, based on more than twenty years of helping establish the video games business, on how our burgeoning industry can continue its breathtaking pace of growth but at the same time be built to last. Famously citing the “Gap of Disappointment” — the space between expected and actual industry adoption of new technology — he admonished us to not let our enthusiasm get ahead of the reality… and somehow managed not be a buzzkill delivering that message. Expect more wisdom like this from JR to kick off the proceedings, and business insights from investors and decision makers.

Beyond JR, the keynote lineup couldn’t be more eclectic. Evolutionary biology will be sharing the stage with the Super Bowl, space exploration, eSports, and the reinvention of the comic book. (You know, all the food groups.) Forty-plus sessions will cover investment, business models, developer war stories, location-based entertainment, mobile, input models, rendering, game design, storytelling, web tech, video production, enterprise use cases… a gamut.

Seems scattered, but there is a common thread: with so much XR happening, we wanted to highlight the stuff that inspires. As we begin to move from demos to commercial delivery, and from theoretical business models to actual ones, this is all getting very real very fast. We know XR is here to stay, and mass adoption is just a matter of time. So rather than focus on the realities of today, Vision is about looking forward without limits. While the show content is rooted in hands-on, deployed projects, we indexed heavily toward works being created by people with an eye to the future.

The last year has seen so much technical innovation in XR, with Unity in the middle of a lot of it. Arguably we’re the leading XR development platform, with two thirds of the world’s immersive content being created using our tools. In that time we’ve added support for several XR devices, and some key rendering and authoring features. So you can expect to find plenty of technical red meat at Vision from us and our key partners, including sessions on interaction design, new hardware, software SDKs, and best practices.

But Unity is at its best, as both a company and a culture, when the tech fades into the background and we let the creations speak for themselves. The entertainment content on display at Vision is breaking ground on many fronts, including interaction design, narrative technique, presence and embodiment, social interaction, and art direction. So you’ll be hearing and seeing a lot from creators using Unity to push the medium forward.

Applications, Applications, Applications

The thing that has me most excited about Vision is the breadth of applications that will be on display. XR is being used in architecture, brand marketing, car design, cinema, education, games, health care, sports, training and toys, to name a few. We’re well past the gee-whiz stage with this technology and are starting to solve real problems. As a lifelong proponent of 3D visualization, I am personally gratified; but more importantly this bodes well for the industry. Solving real problems means making money, and making money means we’ll be around for the long haul.

Now, this situation presents certain challenges to a company like Unity. As the leading XR development platform, we’d love to power all of the world’s great immersive content. But our roots are in game development, and Unity is still by and large a game development platform. Today we don’t have all the pipeline tools and back end services customers required for many non-gaming use cases. Our creators often have to rely on third-party solutions or roll their own to fill the gaps. On the other hand, we are investing heavily in this area going forward. At Vision you will see the serious advances we’ve made to enable Cinematic VR, our next vertical. And that’s just our first foray into XR beyond games.

A Shared Adventure

Though it’s grounded in today, Vision Summit is really about the future. It’s safe to say that we have no idea what this is going to look like in a few years. Which devices are going to win? Nobody knows. Is the mass consumer XR experience going to ultimately be VR, AR or MR, or a hybrid? We literally have no clue. Which will be the killer apps? We’ll only know that when we’re looking at them in the rear view mirror.

In some sense we don’t care about these specifics, because it’s not about the destination; it’s about the journey. Unity created Vision as a gathering place for the industry’s brightest and most creative to explore the potential of immersive computing. Vision isn’t a show, but a summit; not an exposition, but a shared exploration of how this new medium is changing the way we create, play, work and communicate globally.

I hope to see you there. I am really looking forward to the random conversations that are bound to happen when you put this many high energy folks in a concentrated environment. Sharing knowledge and swapping stories. Learning and inspiring. Together.

What is Reality?

Transcript of my keynote speech at the fourth annual Silicon Valley VR Conference.

Good morning. It’s a delight to see friends old and new at the fourth annual SVVR. For many of us, this is where the journey in virtual reality began. A small group of dreamers, holding fast to a future vision based on half-working technology whose success was by no means assured. From the first meetups and the Oculus Kickstarter, to peak hype, to where we are today, it’s been a ride.

Back in 2012, I tried on a DK1 for the first time. I gutted it out for 10 minutes, then took it off, woozily declaring: “not ready.” Too big, too bulky, and decidedly not cool. A veteran of the field, I stayed close to it, cautiously optimistic — but battle-scarred enough to not go all in. Like many of us, the Facebook Oculus acquisition was the watershed event that got even the most skeptical and wizened of us to pay attention. Maybe this time, I began to think, consumer VR could actually work — backed by a communications giant with ample resources and ambitious growth plans.

Just a few months later, I was invited by Karl Krantz to speak at the first-ever SVVR conference. By this point, I had decided, I was all-in: VR, make or break; this time, it’s going to work. I began to cook up the first of several startup ideas, and started advising and investing in companies. And stayed on the front lines as our young industry began to take shape.

And now here we are, three years later. The crazy bet seems to be paying off. VR is going strong, showing signs of continued growth, a diversity of uses and even the glimmer of the first killer apps. We have a thriving ecosystem of hardware, software, tools, big companies and startups, studios, independent creators, and enterprises continuing to push the bounds of what is possible with this evolving technology. We even have career specialists in something called “VR/AR Strategy…” whatever that is.

With this background in mind, and with another exciting SVVR conference about to kick into gear, I thought it would be good to step back and take a look at not only where we are, but what we are doing. And where we can go from here.

By the numbers: where we are is 5 million Gear VRs. 1 million PS VRs. Healthy sales of the desktop systems in the hundreds of thousands. Job Simulator on Steam grossing $3M. VR hitting the mainstream consciousness appearing in TV shows, commercials during sports playoffs, and my favorite VR image of all time:

Here’s 44…. deep in the Metaverse.

And so, it would appear, we have arrived. But have we actually crossed the chasm, from enthusiasm and early adoption on the way to mainstream use? Only time will tell. I don’t have a crystal ball and god knows I’ve been wrong before.

At my company, Unity, we’re bullish on the long term but quite conservative on the short term. We believe that investments in developing immersive content will pay off eventually. But we try not to let our enthusiasm for the potential get ahead of the reality. We are working in lockstep with VR creators in many industries beyond gaming and entertainment. Our struggle are your struggles. We’ll learn together and with luck, prosper together. In a moment, I’ll share some great stories, and insights we’ve gathered thus far.

But first, I would like us all to ponder a question. In the wake of recent geopolitical developments, and the pivotal role technology played in their unfolding, I have found myself asking on more than one occasion:

What is reality?

What is reality, when burst fire laid down in rounds of 140 characters at a time can challenge our most trusted, time-tested institutions, and lies, no matter how brazen, if told repeatedly, can get a major segment of the population questioning the very nature of what is real?

What I tell you three times is true. The neurolinguistic programmers architecting today’s dark revolution know this all too well, and are using it to their advantage.

If something like that can be unleashed using crude tools like social networks and cable TV, what can be done with a medium as powerful as virtual reality? When we can not just bend words and pictures, but all of our senses?

What will happen when we can use VR and haptics to deliver messages straight to the hypothalamus — even implanting memories? Obviously we don’t need to worry about this at scale yet… there aren’t that many Vives in the wild, and the experiences are opt-in and high touch. Nobody is going to voluntarily download a Steam app that turns their world upside down.

But imagine a near future — and I think most of us do — where the VR stack is ubiquitous, deep experiences are commonplace, and we’re plugged in hundreds of times a day the way we are with our phones. We’ll not only work and play in VR — or at that point more likely XR — but we’ll be communicating, socializing and getting our information this way. In this future, VR is the Web, the Web is VR, and the Metaverse is a machine not just for transmitting memes, but entire experiences. Straight to the brain.With much lower friction — see it on your feed and jump right in. Join our clan, hunt and kill those people… because they’re not real… right? Or even if they are real, it’s OK because they’re not us; they’re the *other*.

We’re all familiar with the concept that technology is value neutral: it can be used for good or ill. This holds true for virtual reality of course. But what is happening now in the world is a stark admonition. What we do next matters. A lot. In the months and years ahead, we have a dual responsibility: to grow a young industry, and to use every tool at our disposal to make the world a better place.

And now that I’ve gotten *that* off my chest…

Virtual reality — as opposed to the other kind — gives us new tools for improving our lot. I’m not talking about head-mounted displays, motion tracking hardware, stereoscopic rendering, haptic inputs, machine learning and all the other technological trappings. Those are the enablers, the platform tech underlying this new medium.

No, the tools I’m talking about are communication tools: empathy, escape, embodiment, and engagement. We can use these tools to design immersive experiences that have lasting impact… maybe not solving the world’s problems but at least making positive change.

Let’s explore these now.

Empathy

Much has been made about the power of VR to create empathy.

In a TED talk that launched a thousand startups, Chris Milk famously called VR “the Ultimate Empathy Machine,” citing the medium’s ability to change our perceptions of each other, and to close the gap between the story and the viewer. With television, computers, and phones, there is a space between you and the action. VR takes the story off the screen and into your own eyes: you’re not just a bystander to the ongoing conflict in Syria; you’re experiencing it firsthand. With VR, there is no gap: you’re in the story. You’re no longer a viewer, you’re a witness. You’re Within.

Emblematic Group, Empathetic Media, and other dedicated new media journalists use real-time CG to tell documentary stories in VR. These stories of war, family conflict and abuse of authority make us uncomfortable in a way that no lean-back 2D experience could, by placing us at the center of the action.

In *The Price of Freedom*, you are the protagonist.

Construct Studio takes empathy a step further — completely obliterating the division between the participant and the story. In The Price of Freedom, you are the protagonist in a spy thriller based on the real world drama of cold-war espionage and mind control experiments. Spoiler Alert: it does not end well, and you may never think about our world the same way again.

Empathy helps us understand each other, and our world, in a deeper way.

Escape

If you’re like me, your capacity for self-enlightenment only takes you so far. It’s important to understand the world around us, but sometimes you just want to escape. Books, movies, TV, board games, puzzles video games… we use our communication media to thrill and delight, to tell stories of a brighter future, or sometimes, just to allow us to get off this rock for an hour or two.

Baobab Studios’ ASTEROIDS continues the space-bound adventures of Mac and Cheese. In this version, you play the part of a robot servant, helping the heroes through various tasks and, ultimately, saving the day. ASTEROIDS is feature-film quality computer graphics, but in real time running on a PC with an Oculus Rift, and YOU are at the center of the action. ASTEROIDS just might be the start of the interactive VR story for real. And it’s a delight to behold.

Dreams of “O” — a timeless fantasy world.

Felix and Paul’s DREAMS OF “O” is without a doubt the best 360 piece I have seen to date, based on Cirque du Soleil’s breathtaking aquatic show, “O.” For more than ten minutes, I was taken through a haunting world of water, fire and acrobatics. The tour guide: an white-faced bishop waving a brazier way too close to my face. I was terrified and mesmerized. I lost myself. No distance. No time. Just pure wonder.

The best VR entertainment creates a new world for us to inhabit and takes us on a journey through it… taking to worlds we didn’t even know existed. VR can bring us out of the mundane, even for just a short time. As much as VR is an empathy machine, I think it could also be the ultimate escape pod. And we could all use some of that… maybe now more than ever.

Embodiment

When we talk about VR, we talk a lot about presence. To date, the practice of creating presence in VR has been about establishing a sense of place that you feel is real… but the sense of self within that place hasn’t been there. So while you’re a witness, embedded in the action, it doesn’t really feel there’s a you there. Most VR feels like a third person experience, and creators who have tried to create a sense of self have often failed — mostly because we still don’t have good ways to represent our own bodies.

Life of Us, from Within, takes a novel approach this problem. It doesn’t give you a body; it gives you several bodies.

In this story of all life on earth, you evolve from protozoa, to a fish, to a primate, to human and even post-human. Life of Us uses positional tracking to great effect — you are essentially on rails so you don’t need to figure out how to move — and gives you enough agency with touch controllers

so that you could flap your arms to fly, wave to your companion, and draw streams of light in the sky.

It’s a totally great and fun ride, but where Life of Us really blows the lid off is in taking presence to a whole new level. It provides a live companion on the journey, so that we can feel ourselves through another. This goes beyond presence to embodiment. By taking the guesswork out of operating your body, and giving you someone else to see yourself through, you feel like you’re not only there, but inhabiting a true self. It’s a breakthrough in interaction design, and may just point the way for future applications in storytelling, education, training and social interaction.

Engagement

VR is already being used to market and sell products. And it’s going to be used to do that in the future. A *lot*.

As will AR and every other R. Every inch of available virtual real estate will be up for sale for manufacturers and marketers to hawk their wares.

The history of advertising is, after all, the history of a rise in production value. And with VR we so much more production value at our disposal: 3D graphics, animation, 3D sound, and haptic inputs, to name a few.

Immersive technology is going to be the ultimate marketing tool, because all of these goodies enable us to fully engage the user in an experience with all the senses, and allow them to actively participate. UK-based REWIND Studios recently created a multi-sensory, multi-user experience that puts you in the driving seat of a Formula E racing car, taking you on a thrill ride and highlighting the vehicle’s new features. Had fun? Good… maybe you’ll buy the car. But even if you don’t, your opinion of that manufacturer most likely went way up, and so did their brand.

All these deep interactions also result in way more data. Beyond the conscious interactive choices that the user makes, we can monitor direct manipulation input, track the head and eyes, and analyze sentiment based on subtle movements. With immersive technology, we’ll glean insights into individual and collective behavior even more than before. We can design better products, sell more of them, and better satisfy customers.

Now, these same techniques can be used for a lot more than sales and marketing. Engagement is key to the future of learning. The more actively involved we are, and the more physically engaged, the more we retain.

Everything from education, to situational awareness training, to operating machinery is going to get a huge bump from immersive technology. And the data that can be collected will help the educators and trainers develop better materials, in a virtuous circle.

The Future

These four tools — empathy, escape, embodiment, and engagement — give us a broad base to work from, to design immersive experiences that have lasting impact. They can be used to tell stories, about the real world or made up worlds; to teach; to ease pain and suffering; to sell products and build a deeper relationship with customers; to better relate to each other and the world around us.

VR is the most powerful medium we have devised to date. It can be used for good or ill, but in the balance I’m hoping it’s the former, and that collectively we can help safeguard our future. As technologists, scientists, educators, designers, storytellers, and entertainers, we do what we do to have fun, and to make money, but ultimately we do it to change the world… for the better.

Thank you and enjoy the show.

Getting Real

2017 Is Going to be a Watershed Year for Cinematic VR

I had always wanted to go to the Sundance Film Festival, but somehow, year after year, the fates conspired to keep me away. Lucky me, now I get to go as part of my day job at Unity. Our team came out in force to be part of this year’s great unveiling of virtual and augmented reality at New Frontier, the festival’s initiative at the convergence of film, media and technology. Unity’s technology powers most of the world’s XR, so we came to support our partners creating great content, and highlight the best of the best via our Made With Unity marketing program.

First off… what a difference a year makes. If you look back only one year to New Frontier 2016, we’ve gone from plopping a camera inside a circle of people to effectively using 360 with full sets and multiple cameras; from tinkering with CG set design to creating full worlds; from exploring the basic interaction to designing intuitive controls to enable true agency; and from establishing presence to enabling embodiment. We’ve even thrown augmented reality into the mix and are seeing an artful blend of animation, story and physical space. The content being showcased here at New Frontier has kicked it up a notch in so many ways that to me, 2017 is a watershed year for cinematic VR, and it’s only going to get better from here.

As with any new medium, the first few years see us trying to replicate old patterns — like those first attempts at TV, which just put a camera in front of a stage play. We know those are transient states, and we don’t know exactly where we’re headed and what techniques will stick. And we have no benefit of hindsight. It reminds me of what Henry Ford said when when developing the automobile. He famously remarked, “If I had asked people what they wanted, they would have said ’faster horses.’” The implications of a revolutionary new invention — like virtual reality — can’t be fully appreciated until long after they are first introduced. But the good news is, we’re past the prototype stage, and we’ve rolled out those first model T’s. So let’s take a look at a few things that are happening to push the envelope of Cinematic VR.

The image at the top is from ASTEROIDS!, Baobab Studios’ Made with Unity sequel to last year’s Invasion! ASTEROIDS continues the space-bound adventures of Mac and Cheese. In this version, you play the part of a robot servant, helping the heroes through various tasks and, ultimately, saving the day. ASTEROIDS is feature-film quality computer graphics, but in real time running on a PC with an Oculus Rift, and YOU are at the center of the action. That’s a long way from a year ago, when most of the graphics we’d see in VR looked more like mobile phone game graphics– dumbed down to meet the VR’s rigorous performance requirements to maintain frame rate and a comfortable experience.

Presence in VR just took a great leap forward with Life of Us, the new piece from Chris Milk and Within — also Made with Unity. Life of Us is a multiplayer experience that takes you on a journey of evolution, from a single celled organism, through higher forms of life, to humanity and beyond. Using positional tracking and touch controls, you inhabit these life forms and travel through space with a companion. In my case, my player companion was my wife, who was in town for the festival. We swam together as fish, flew like pterodactyls and danced as beautiful shining robots.

Life of Us used positional tracking to great effect — you are essentially on rails so you don’t need to figure out how to move — and gave you enough agency with touch controllers so that you could flap your arms to fly, wave to your companion, and draw streams of light in the sky. It’s a totally great and fun ride, but where Life of Us really blew the lid off was in taking presence to a whole new level. To date, the practice of creating presence in VR has been about establishing a sense of place that you feel was real… but the sense of self within that place hasn’t been there. Things mostly feel like a third person experience, and people who have tried to create a sense of self have often failed, because we still don’t have good ways to represent our own bodies. Life of Us dealt with that in a novel way: by providing a live companion on the journey, we could feel ourselves through another. This goes beyond presence to embodiment: I felt like I was not just there, but inhabiting a true self.

At the other end of the spectrum entirely is a piece called Zero Days, by Brooklyn-based Scatter. Based on a documentary film of the same name,Zero Days uses immersive VR to tell the story of the computer virus Stuxnet, used during a clandestine mission by the U.S. and Israel to sabotage an underground Iranian nuclear facility. Zero Days throws out the VR playbook from last few years: unlike most approaches to VR storytelling, it doesn’t try to give you a sense of self, or presence. And it’s not trying to create a verisimilar world, with a full, rich environment: it is abstract, laying out an information space for telling a linear story. Zero Days brings together traditional media — video, audio, CG animation, text, charts — into a seamless VR presentation, creating a new art form for telling stories. To me, it’s a tour de force of visual presentation — and may just be a glimpse of the future of documentary storytelling.

I’m delighted to say that 360 video has also made great strides. Universal is about to release Fifty Shades Darker : The Masquerade Ball VR Experience for Gear VR. This 360 video trailer features elaborate sets, great camera work, multiple camera angles, and uses the cast from the feature film. It was as high production value as a feature film, but with the intimacy of VR video. I also saw a 360 music video from 1215 Creative director Jenn Duong — giving us a look into the artist Banks, using metaphors that contrast a surreal dream like state with performance art. This is to say nothing of Felix and Paul’s masterpiece, DREAMS OF “O” — without a doubt the best 360 piece I have seen to date — based on Cirque du Soleil’s breathtaking aquatic show, “O.”

In one short year, I think we have come a long way from the “slap a camera in the middle of the room” approach, to a place where video directors are now using the medium to create real art with a point of view.

Finally, augmented reality has come to New Frontier in a big way. There were multiple AR pieces on display, including Heroes, a romantic experience about movement, space, and scale set to the tune of the David Bowie hit song. Heroes moves from VR to AR, beginning with a Gear VR 360 video of live dancers in a theatre, and ending with a Hololens installation featuring animating 3D figures, models of the theatre, and branching narrative. Heroes plays heavily with scale — of characters, environments, models and the room itself.

To sum up, 2017 is a watershed year for cinematic VR, and it’s only going to get better from here. Creators have pushed the envelope on fidelity, presence, abstraction, point of view, and use of scale. These will be the building blocks for VR and AR, moving us from traditional storytelling to the creation of whole new worlds — which is where Brett Leonard of Lawnmower Man fame thinks this is all going… and I’m not about to argue.

I think we’re seeing the first true native VR pieces– not throwbacks to other eras ported to VR– but content that can stand on its own, that you can only do in VR and AR… that people will actually come back to again and again, maybe even pay money for.

Things are getting real up in here…

To learn more about the VR and AR projects a Sundance New Frontier, check out the festival page.

Unity

Today I am thrilled to share that I am embarking on a new chapter: I have joined Unity Technologies as Global Head of VR and AR, overseeing the company’s strategy to expand its business based on the new immersive platforms taking the world by storm.

Now, if you don’t know about Unity, first off… what planet are you living on? Unity is by far the most popular tool for making games and, by extension, VR and AR. It’s accessible and easy to approach for the beginner, but has all the power the pros need. Unity is free for personal use, and has affordable priced tiers above that. It supports all the major platforms. OK, I’ll stop gushing; the list of features and benefits go on and on. But let’s just say I’m such a big fan, I devoted most of my latest book to it.

For almost my entire career in technology, I have been working in 3D graphics. Not gaming per se, but graphics — for scientific visualization, architecture, education, and other non-game uses. I got my first taste in the late 80’s, working on 3D rendering software drivers for CRT displays, before graphical user interface systems were even prevalent.

A lifelong fan of comics, art and design, I have always been fascinated by the rendering of beautiful images and the visual presentation of ideas. When I did a life pivot from working musician to computer programmer, I certainly didn’t expect to be doing something as sexy as graphics; I figured I would be writing financial applications or something else involving wearing a tie to work. But I got lucky, and from early on in my career I have been fortunate enough to be able to write programs that put pretty pictures on the screen.And I never, ever had to wear a tie.

So when I moved from the east coast to the west to pursue my fortune in the new world of the Internet, I guess it just made sense that the first person I connected with was Mark Pesce, a network geek who needed someone to build a 3D interface to the then-nascent World Wide Web, in a way-too-early attempt to bring Neal Stephenson’s Snow Crash to life. We were crazy, and it didn’t really work, but our mad experiment put me on the path that I am still on, exploring themes around building interfaces to the world’s information in VR and democratizing 3D development.

That last bit is key to our story. 3D development is inherently hard. The extra dimension is a bitch, partly because the tools we have on our desk render to flat screens, making interaction and manipulation harder, but mostly, because math. Graphics programming involves a lot of math, and 3D is more math-intensive than 2D, especially given the things we want to do with it, from mesh deformation to lighting to animation to physics.

I will let you in on a deep, dark secret: I hate the math. I love to brag that I got an A+ on my linear algebra final, but well, that was a lifetime ago, and now if I have to decompose a matrix I break into a cold sweat. I still fall prey to the occasional impostor syndrome because of this; but I ultimately take comfort knowing that I have spent the last couple of decades focused way higher up the stack, trying to do good by spreading the Good Word of real-time 3D.

During this journey, I have discovered something interesting: I think a lot of folks in our industry would prefer that 3D development remain a black art.Rather than devise pipelines, engines and tools that make it easier, most of what I have seen in graphics system design has moved in the opposite direction. I was on a mission to bring 3D to the world, but it felt like everybody else was trying to keep it tucked away behind hidden doors that could be unlocked only with secret handshakes.

Imagine if you wanted to build a web page today, but you didn’t have a web browser. You just had the Windows API and an Internet connection. You’d have to build all the other bits up from scratch: page layout, parsing HTML, decoding packets coming over the network. Inconceivable, I know. Well, that’s how game development was for years.

And then came Unity.

I first discovered Unity in the late 00’s, when they helped The Cartoon Network deliver an MMO in a web browser using their plugin. Even though I wasn’t building games at the time, I admired the company’s singular focus on their customers and core business. I soon became friends with founder David Helgason, and from then on I made sure to keep a close watch on the company’s progress. When the VR consumer boom hit, I now had a reason to dive into Unity development, and began using it for my own work. Simply put, it blew me away.

I have tried many 3D development tools over the years, and for the most part they, well — they just suck. The learning curves are really steep, and the user interfaces are designed abysmally. Unity, on the other hand, was a breath of fresh air: it made game development accessible, even to non-gamers like me, with a clear set of concepts and an approachable interface.

One could argue that it’s not necessary to provide easy tools for 3D game development, because game developers are highly technical and don’t need that much hand-holding. For hardcore games, the point is debatable. But VR and AR are a whole different beast. A large percentage of would-be VR developers aren’t graphics experts. They are filmmakers, educators, psychologists, and domain experts in various fields outside of computer science. They need easier tools, ones that take the rocket science out of authoring by taking care of the hard aspects of 3D. They need Unity.

Doubly awesome about Unity is that it also has a ton of power. The company has invested significantly in rendering, animation, physics and other areas, making it a professional-grade system for high end use as well. So developers can get started easily, and continue to use the product as they grow professionally. Some will go on to build mind-blowing creations, tomorrow’s VR blockbusters.

Unity has built the world’s largest base of 3D creators on the planet by following three core principles:

Democratize development.
Solve hard problems.
Enable success.

Unity has been doing these three things for years for the game industry. And since that has gone so well, the company is now set up to be a dominant player in VR and AR development. Where this will go is anybody’s guess. As we all know, it’s very early for VR and AR. The rules have yet to be written about what makes for good immersive design. The hardware platforms are still shaking out, with no clear winners. The genres are up for grabs, and we have only a handful of killer apps. There’s still so much to do!

And now I’m here to help. Let’s do this.

It Takes a Village

The W3C VR Workshop team. See us in full 360.

After all this time, VR really is the Web’s Next Big Thing. A two-day workshop brought the Web’s best and brightest together to define the future of the medium.

TL;DR: The web community’s next big push on browser technology puts VR and AR front and center. The Immersive Web is going to happen, and sooner than we think. It will take vision, hard work, compromise, and lots of really smart people.

The New Normal

I’m not really sure how it happened, but within two years WebVR — the technology that connects web browsers to virtual reality hardware — has gone from a wacky idea to the new normal.

WebVR was hatched a few years back by WebGL creator Vlad Vukićević of Mozilla, collaborating with Google Chrome hotshot Brandon Jones, and publicly demoed for the first time at our San Francisco WebGL meetup in the summer of 2014. The demos were crude, and the tracking sucked, but it was a start, and we were excited by the possibilities.

In the year that followed, WebVR was greeted with blank stares by the VR hipster elite and outright derision by industry players. We heard familiar arguments: bad performance, because JavaScript; too much bureaucracy with standards groups; and anyway, why would you want to do something like that? To be fair, folks had a lot on their plate, and most people didn’t have the mental bandwidth think about something as abstract and forward-looking as WebVR.

But over the past year, something changed. Seed-stage investors started asking me about WebVR, wanting to know the lay of the land. VentureBeat sang its praises and touted the benefits. Microsoft announced support for WebVR in Edge and contributed to the spec to make it suitable for AR. And in a penny-drop moment, a team at Oculus approached me with their plan to get into the WebVR game in a big way. This led to WebVR being featured in the keynote at Oculus Connect 3, with a demo by my fledgling startup and WebVR prominently placed in the Oculus Developer Portal.

It appears that WebVR has arrived. In a previous post I went into some detail about why I think the timing is now. But that only covers the why. How we are going get there has been by no means clear, till now.

The Final Frontier

While all this WebVR goodness was brewing, a group from the World Wide Web Consortium (W3C), the people that brought you the Web, organized a Workshop on Web & Virtual Reality. The workshop, hosted by Samsung in San Jose, CA on October 19th and 20th, brought together browser makers, content creators, application developers, and technical experts to discuss enabling technologies and standards for an open VR infrastructure.

The very existence of this meeting showed that the powers-that-be in W3C understand the importance of VR as the new frontier of web development. That’s a big win, in and of itself. But the reality of the event went beyond that. The technical quality of the presentations, the concrete plans shared by product vendors, and the positive energy and spirit of collaboration showed how seriously the industry is taking this initiative. In my view, this workshop was another watershed, in a watershed month for VR that included Daydream View, PlayStation VR and Oculus Connect 3.

The W3C Workshop covered broad ground and when it could, went deep. After two days of lightning talks, panel sessions and breakouts, my ears were fairly bleeding from the information overload. I am sure the organizers will post full meeting notes soon. (See the schedule page for the detailed list of speakers and topics.) In the meantime, here are some highlights.

Sean White keynote. An old friend from the VRML days, now VP of Technology Strategy at Mozilla, Sean White delivered a homey keynote that hearkened back to early VR and the collaboration that built the web, setting the tone for the rest of the workshop.
WebVR API Update and Browser Support. There are more tweaks coming in the WebVR APIs, leading to a likely 1.2 version before broad adoption. Google, Mozilla, Samsung and Oculus shared concrete plans and expected ship dates for desktop and mobile VR browsers.
Lightning talks. A barrage of 5-minute lightning talks covered UI design, accessibility, 360 video formats and metadata, immersive audio, multi-user networking, and declarative VR languages and 3D file formats.
Breakout sessions. We split the group into sessions on various topics, including high performance VR implementations, hyperlinking in VR, and extending the browser DOM into 3D.

My fingerprints can be seen on a lot of the Workshop’s content, but I take particular pride in one development that I hadn’t even planned for. glTF, the new file format standard for 3D scenes, continues to build steam, with a groundswell of endorsement from industry partners over the last several months. glTF figured prominently in many discussions over the two days. Folks even floated the idea of glTF as a built-in format that browsers could natively read and display (analogous to JPEG images for 2D), with immediate application as 3D favicons, hyperlink transition animations, and built-in layer graphics, e.g. for splash screens and heads-up displays. Whoa. Mind blown.

Rip Van VRML

As fun a time as this was for me, it was at times surreal. A generation of brilliant technologists had locked themselves in meeting rooms to design the Metaverse, along the way rehashing ideas we explored two decades before, such as 3D scene representation, VR interface design, and shared presence. In earnestness and with great enthusiasm, the kids at this workshop were reinventing wheels left and right. But how could they know? Many of them were in middle school the first time around… if they were even born yet.

A modern day Rip Van Winkle, I had fallen asleep during a 1996 VRML meeting, and woke up twenty years later in the same room. Then I began to realize that things were different. People were holding these little computers in their hands. The Internet was fast and, apparently, connected everybody on the planet. And you could fit VR equipment in your backpack! Most of all, the people leading the discussions weren’t crazy futurists on the fringe; they were sane futurists working at mainstream companies.

It was 2016, and the real world was ready for the virtual one. While most of the problems are the same as twenty years ago, now we’re looking at them through a post-Facebook, post-Pokemon Go lens, and building on technology that is thousands of times more powerful.

A Community Effort

The W3C Workshop explored a vast landscape of technologies and standards, interface design and best practices, and tools and frameworks. While it was a great kickoff, it was just that: a kickoff. There will be much hard work going forward.

We already have a head start, and some success under our belt. WebVR is maturing and really working now, with 90FPS head tracking and support in many browsers. glTF is a ratified standard from Khronos, with steadily growing support. Much of what we discussed at the workshop simply extends existing standards like HTML Video and Web Audio. So we’re not tackling any of this from a standing start, or from an ivory tower. The output from the workshop will be brought back to working groups at W3C and Khronos, or new groups will form to tackle new pieces of the problem.

That, generally, is how the process will unfold. But it’s not just about process; it’s about people. People were the key to the success of this workshop. The organizers, Dominique Hazael-Massieux, Anssi Kostiainen, and Chris Van Wiemeersch, worked tirelessly to put on a first-class event, extremely well-run with top-notch content. The familiar names of WebGL lore — Josh Carpenter, Brandon Jones, Michael Blix and Laszlo Gombos — have been joined by new ones, like Justin Rogers and Amber Roy of Oculus, Nell Waliczek and Adalberto Foresti of Microsoft, Ada Rose of Samsung, Kevin Ngo and the A-Frame team from Mozilla, and Shannon Norrell and other energetic community-builders. There were numerous positive contributions and, given the headiness of the subject matter, the mood remained light throughout the proceedings. There was a real spirit of cooperation and hope for the future.

If we can bring a fraction of this energy to bear in the coming months, we will make great progress. The movement is growing. We have enough people on this, with big brains, pure motivations and a shared vision. And that’s good… because it takes a village to build a Metaverse.

Third Time’s the Charm

How VR and the Web Have Finally Converged — In My Lifetime

Twenty five years after the first consumer VR crash, virtual reality is poised to upend human-computer interaction, the Internet has disrupted every facet of life as we knew it back then, and the two are now on a collision course. The Metaverse that we’ve all been dreaming about for decades — that shared vision of everybody connected and communicating in a web of virtual reality— is upon us.

Twice before, the industry attempted to consummate this chemical wedding… and twice now, somebody ran from the altar. Well, this time, the bride and groom are pure of heart and truly ready. The confluence of cheap VR hardware, accessible 3D development, and ubiquitous networking has set the stage for an explosion of VR content, delivered over the World Wide Web.

Why has it taken so long? And why is it happening now? I’ll tell you. But first, a little history.

The Best of Intentions

The notion of combining VR and the Web is hardly new. Tim Berners-Lee put out a request for proposals way back at the first-ever World Wide Web conference in 1994. Mark Pesce and I answered the call; the result was our problem child, VRML.

VRML was designed as a universal language for authoring 3D applications on the web, the first piece of a technology stack intended to bring Neal Stephenson’s vision of the Metaverse to life. After we made 3D rendering universal, we figured we would tackle multi-player networking and then, finally, when a new generation of VR hardware was ready, we’d connect it all together — with the assumption that these other pieces were right around the corner.

VRML was built on then-state-of-the-art tech: an open, scalable 3D infrastructure allowing anyone to create and share, with the burgeoning Internet as its backbone. We created VRML out in the open, didn’t patent anything, and gave everything away in the hope of starting an explosion of 3D creation online.

VRML captured the imagination of the fledgling Web industry. Software leaders Netscape, Microsoft and Adobe hooked up with hardware titans like Silicon Graphics, Sun, Intel, Apple and IBM to build our collective 3D future. Though a few had knives under the table, most of the big guys did their best to cooperate on standards. We created killer demo showcases. The hype train, powered by SGI’s marketing machine, kicked into high gear. Startup fortunes were made.

But there was a problem: we didn’t have a market. The processing power and bandwidth required for quality 3D weren’t in the average home. Most people with PCs didn’t even have a Web browser yet. VRML was a noble experiment, conceived with the best of intentions, that ultimately came up short. Wrong place; wrong time.

Worlds in Collision

In the 2000’s, virtual worlds like Second Life promised us the Metaverse all over again via real-time, user-generated 3D social environments, running on a new generation of cheap high-performance PCs. Second Life was, for its time, a damn good experience, much better than anything ever built in VRML.

By 2007, Second Life was at the center of its own hype bubble, hitting the cover of Business Week with the promise of a new way to play and communicate, and even make money by selling each other virtual stuff. SL gave rise to well-funded copycat startups, including one that I founded. But by 2010, most of the virtual worlds companies from that period were gone. Second Life was the sole exception, having built a solid business, though not a large one by Internet standards.

So why did the category fail?

Partly, it was because of the lack of scalability inherent in such a closed system. While VRML approached the Metaverse bottoms-up, via an open infrastructure and industry cooperation, these virtual worlds systems went at it top-down, delivering highly structured and stylized experiences, via AOL-style walled garden networks. The products provided powerful authoring for users to create their own content — but each company owned its own full stack, from client to tool to server. Without an open ecosystem it is really hard to achieve Web scale, and it’s all on the shoulders of one company to deliver continual value and growth.

Second Life and its ilk may have also floundered because open-ended worlds are inherently limited in what they have to offer compared to their achievement-oriented, MMOG cousins like Warcraft and League of Legends. It takes a lot of commitment to build a Second Life, and for many, it’s apparently not worth it, because there isn’t a big reward at the end.

But mostly, I think virtual world growth stalled because it got sideswiped by something bigger. Social networks provided 80 percent of the bang for way less effort, and ran everywhere, without needing to install custom software. Why go to the trouble of buying a gaming PC, installing a fat software package, and learning how to build 3D worlds, when you can instead sign into Farmville with a click using your Facebook ID, and grow virtual soybeans on your shiny silver Macbook?

Move along; this is not the Metaverse you’re looking for.

An Idea Whose Time Has Come

Half a decade after the virtual worlds bubble burst, everything has changed. Today’s smartphones have way more 3D power than the workstations that originally ran VRML; everybody is connected on fast networks; and affordable consumer VR hardware is blowing up. If now isn’t Metaverse time, I don’t know when is.

A significant development got buried in the noise around the resurgence of consumer VR. 3D is on the Web to stay, with the advent of WebGL. WebGL makes it possible to deliver hardware-accelerated 3D in a browser, programmed in JavaScript and accessible to everyone — with no additional download. WebGL runs on all desktop computers and modern smartphones. At 3 billion seats and counting, it’s ubiquitous.

But thus far, with a few exceptions WebGL has been an optional add-on to commercial sites. In the end, the results are still rendered on a flat desktop or mobile screen — granted, with more speed and sizzle, but still part of a 2D experience. Well, with a stereoscopic VR or AR display, that’s not an option: you must render in 3D. So if you want to create a Web-based application for VR, you really have no choice but to use WebGL.

Now, if you’re reading this, I am just going to assume that I don’t need to convince you why people want to create VR for the Web. To me, the idea that there won’t be VR applications built on Web tech, based on Web content, well… that’s just absurd. It’s just as absurd as someone in 2007 predicting that smartphones wouldn’t deliver Web content, and mobile apps wouldn’t someday be mostly based on HTML. Well, in 2016 they do, and they are.

Market factors will force the industry’s hand on this. The desire for cheaper, easier ways to produce, deploy and deliver VR is there: not everyone can master a game engine, and labor through the deployment and maintenance process that comes with app packaging and app store distribution. And for consumers, the long tail of applications demands an open system without the friction of app store discovery, download and installation. The makers of VR hardware used the mobile app store model as the starting point to the get the industry kick-started, but surely this is just a transient stage on the way to a fully connected Metaverse.

The technology underpinnings are now in place. In addition to WebGL, we have WebVR, a new set of VR browser APIs in development since the ink was drying on the Facebook/Oculus acquisition. We also have glTF, the new portable 3D file format that is like JPEG, but for 3D. Add to these myriad JavaScript libraries for creating VR, and the Electron framework for building native apps in HTML5, and the sky is the limit. These pieces are the kindling for a wildfire; all it’s going to take are a few simple tools and killer apps to set it off.

Which tools and killer apps? We don’t know… and we don’t care. The Metaverse may have been imagined in fiction as the product of a singular vision, a Grand Architect of the Universe mapping out how, when, why and where people will be interacting socially in VR. But that’s not how it will actually get built. The Metaverse is going to be a messy, out-of-control affair, with multiple entry points, and a face and shape that we can’t yet imagine. What we do know is that it will be comprised of a billion plus people using VR systems connected via the Web. That’s all we know. And that is enough.

Don’t take my word for any of this. Google and Mozilla are leading the way by implementing VR-enabled browsers, but you will be hearing shortly from others that are going to spearhead the effort with market applications, enabling platforms and distribution networks. The dynamics I have described here aren’t just based on two decades of my own hit-or-miss insights; they are rooted in real market pain points, developer desires, and stated strategy from big industry players.

It’s finally here.

I guess third time really is the charm.

Unfettered

Every year or so I touch base with the next generation of multimedia developers at Carnegie-Mellon’s Entertainment Technology Center, located at Electronic Arts’ Redwood Shores campus. ETC is the premier graduate program for interactive entertainment, a multidisciplinary endeavor run by my friend and mentor Carl Rosendahl (yeah that Carl as in PDI as in Shrek). Yesterday I made the pilgrimage to ETC, and the visit did not disappoint. I was treated to two excellent projects by the students.

One of the projects was an experiment in untethered virtual reality called Project Gotan (not to be confused with Gotan Project, the electronic tango band). Project Gotan is an experiment in virtual reality that puts a Durovis Dive 7 together with a Project Tango tablet, allowing you to freely navigate room-scale VR using a mobile device. The Tango continually scans the physical environment, creating a virtual world in real time to represent your actual surroundings. The upshot of this is that you can be in a fully immersive headset, optically cut off from the outside world, and yet walk around freely without fear of bumping into things and, more importantly, without wires and a cable sherpa keeping you safe.

That’s me, all over: walking around in room scale VR with no wires.

A nice touch was the procedurally generated, voxel-rendered world in the style of Minecraft. Unlike the game, this world doesn’t need to calculate surface areas using marching cubes or some other computationally challenging task on a tablet. The Tango is delivering point cloud data, which Gotan can render directly as those familiar grass-and-dirt block shapes without much fuss; this world had a few million points in it, easy.

The world of Project Gotan, with real-world physical obstacles rendered as voxels in the virtual world.

One small critique I had was that I thought the size of the voxels was deceptive. In Minecraft it feels to me like voxels are about 1/2 meter on a side each. Now this is the interesting part: even though I have only ever experienced Minecraft on a flat screen (hope that changes soon!), I have an unconscious sense of the physical size of a voxel. I brought this innate assumption with me into the Project Gotan environment, and during my first forays it tripped me up, literally and figuratively. The Gotan voxels seemed to be just a few inches on a side, which made me move very tentatively through the space until I got accommodated to the difference in scale. Don’t get me wrong; this is a minor complaint, and I only share the story for the observation about how our sensory systems are intertwined with muscle memory — crazy!

Even though I have only ever experienced Minecraft on a flat screen, I have an unconscious sense of the physical size of a voxel.

Project Gotan is a grad student project, an experiment to see how far we can push room-scale immersion. It’s still crude — but maybe it points to where we can take VR with a little imagination and a lot of hard work jamming for a grade. This was Unfettered VR and now that I’ve had a taste, well, the bar has been raised for me.