As the roar of the metaverse hype machine fades to a ho hum, there’s growing evidence that what can be called a “metaversal” approach to production is starting to take hold in the broadcast industry.
Paralleling the headline-grabbing and often flawed efforts to deliver immersive virtual reality (VR) experiences with sports programming, broadcasters have quietly been learning how to leverage volumetric VR and augmented reality (AR) production tools in their workflows. This has allowed them to add compelling graphics dynamism and personalized viewing experiences to internet-streamed 2D renderings of news, sports, and other live broadcasts that are reaching viewers worldwide.
At the same time, by bringing these solutions into the production mainstream, broadcasters are getting the hands-on experiences in volumetric productions that are essential to producing a new generation of 3D immersive viewing experiences. These efforts are gaining currency with ongoing improvements in extended reality (XR) hardware and software.
In the Real World
Notably, while many high-profile sports and other VR programming initiatives were done in by poor viewing experiences, some providers, including the NBA—in a recently renewed multiyear deal with Meta; and NBC with its live VR streams during the 2022 Winter Olympics—have stayed the course. With these new productions they’re demonstrating that it’s possible to eliminate pixelation on VR headset screens, stomach-churning delays in responses to shifts in viewers’ fields of vision, and other impediments to user adoption.
Equally important to the improvements in consumer experiences are major advances in headgear, including smaller form factors, tetherless VR head-mounted devices (HMDs) that free users from wired links to computers, and eye movement-tracking sensors that lead to more accurate renderings of what users are looking at.
Looking forward, while Apple’s Vision Pro mixed reality (MR) HMD set for launch in early 2024 with a starting price tag of $3,499 is targeted to the enterprise and the high-end consumer markets, its incorporation of a dazzling array of advances combining AR and VR technologies exemplifies where mass market user experience is headed as prices for such devices fall over the next few years.
The new volumetric production-driven perspective on the broadcast industry’s migration into the metaverse runs counter to the current sense of disarray in the metaverse camp, which may mean new terminology is needed to avoid detracting from the transformative developments now underway.
Notably, Apple CEO Tim Cook never used the word “metaverse” when he introduced the Vision Pro in June. Indeed, with Meta’s Reality Labs laying people off in droves after racking up a $13.7-billion loss in 2022 and headlines trumpeting the demise of metaverse projects like Microsoft’s AltspaceVR, Walmart’s Universe of Play, and Disney’s Next Generation Storytelling & Consumer Experiences, there’s far less of the metaverse-infused banter that saturated the internet two years ago.
Building a Foundation
But no matter how the next phase of engagement with internet-based technology is labeled, there’s no denying that the broadcast industry, along with many other industries is moving inexorably toward building a foundation to operating in a new era blending real-world experiences with immersive virtualization. The pace of XR adoption across all sectors is reflected in many recent research reports.
For example, MarketsandMarkets predicts the XR market—encompassing devices, software, and applications—will grow from $40.1 billion in 2023 at a compound annual growth rate (CAGR) of 22.7% to reach $111.5 billion by 2028.
Researcher Brainy Insights, citing demand for 3D/360-degree content in sports and other entertainment as well as advertising, forecasts even faster growth in the global volumetric video segment of the market with projected expansion at a 29.3% CAGR from $2.1 billion in 2021 to $21.2 billion by 2030.
The broadcast industry’s embrace of volumetric technology in traditional 2D program production is a welcome development for a company like Arcturus Studios, which since its founding seven years ago has amassed a full suite of solutions, including acquisition of the Microsoft creative platform, that are designed to provide end-to-end support for building and streaming immersive 3D user experiences.
Now there’s an opportunity to apply that technology to address a major pain point impacting broadcasters everywhere, notes Mark Gerberman, vice president of sales at Arcturus. With the amassing of ever more data that can be used to enrich viewing experiences, broadcasters need to make it easier for consumers to absorb and understand all that information.
“A lot of our focus now is in the virtual production space,” Gerberman says. “Humans by nature see things in 3D, so being able to use volumetric technology to more concisely convey the data producers are pulling off the field is top of mind.”
The results of such efforts are on view just about everywhere. Across the globe consumers have grown accustomed to seeing 3D data displays generating stats in sync with the narratives of on-air personalities broadcasting from studios that are often embellished with virtual furnishings and backdrops.
The impact of the volumetric production agenda is noticeable in post production processing, according to Allan McLennan, president of 2G Digital Post, a leading Burbank, Calif.-based post house serving motion picture and TV producers worldwide. “Where we see it is in the sheer size of the files, which are a lot larger than a traditional video capture or film,” McLennan says.
Once news, sports and other live content is produced on location for immediate output, the completed files are sent to post production where they’re processed for on-demand access. “Those files need to be compressed, packaged, secured, and pushed out with a very heavy eye on quality control,” McLennan notes.
2G Digital is known for its QC expertise, which is essential to meeting producers’ specifications for packaging and local distribution, McLennan adds. “There’s always something that has to be looked at for delivery to the different platforms in the marketplace,” he said.
While the bigger files benefitting from volumetric enhancements might add a second or two to the post production processes, today’s state-of-the-art hardware can handle the loads. “What it really comes down to is updating new types of software,” McLennan says. “We’re on top of that on an ongoing basis.”
Deloitte Initiatives
With the growing focus on volumetric production in 2D programming, the industry is peeling back layers of the XR onion, normalizing the use of tools that will take them much farther over time, Gerberman notes. Indeed, as he and others in the field can attest, the migration is well underway with a rapidly expanding spate of initiatives at national and local levels of broadcast operations worldwide.
For example, Allan Cook, managing director at Deloitte Consulting, is deeply involved in such initiatives as head of Deloitte’s Unlimited Reality practice, which has been expanding across multiple industries since its genesis several years ago on the heels of metaverse-oriented projects involving the U.S. Department of Defense. There’s a lot of volumetric production activity underway in the broadcast industry, Cook says.
Cases in point include multi-year Unlimited Reality engagements with the U.S. Golf Association (USGA) and Sinclair Broadcast Group, which, unlike the unit’s other broadcast industry clients, have been willing to publicize what they’re doing with the firm. These applications are based on volumetric 3D production techniques that leverage AI, 3D computing, Web 3.0 technology, and advanced streaming to produce experiences meant for 2D viewing on conventional device screens as well as more immersive 3D viewing experiences if clients want to engage with people through AR and VR eyewear.
One of the features offered with the latest version of the USGA App used in this year’s U.S. Open and U.S. Women’s Open championships allowed viewers to follow play tournament-wide on interactive streams to their PCs, tablets and smartphones. “With our app you could watch any player at any time on any hole from any direction,” Cook says.
“You could watch a player in 3D hit the ball and see exactly where it landed on the fairway,” he adds. On the green, “you could zoom down to the ball going into the hole in near real time.” In addition to shot tracking, the app allowed viewers to tap into scorecard highlights of all televised shots and to receive customized alerts about players they were following.
Such capabilities demonstrate the drawing power of this technology. With tens of thousands of users worldwide, the app registered as a top-five app in the Apple store during the men’s open, Cook says. The results are resonating with the USGA, which this year for the first time has extended use of the app with varying feature sets across all 15 of its championship tournaments.
360-degree Socialized Viewing Environment
The other publicized example of what Deloitte is doing occurred with the Sinclair-owned Tennis Channel’s coverage of this year’s BNP Paribas Open in Indian Wells, CA. The app— offered on an invitation-only basis to viewers worldwide—delivered a multi-faceted 2D user experience based on 3D volumetric production technology supported in part by Epic Games’ Unreal Engine, a leading provider of 3D creation tools for multiplayer video games.
This was the first project undertaken in fulfillment of Sinclair’s goal to build a 360-degree socialized viewing environment with its live sports and possibly other broadcasts. “We created a number of experiences you wouldn’t be able to get in your home or even at the event,” Cook explains.
The app made it possible for users’ avatars to play tennis on a digital replica of the Paribas tournament’s center court. “We also allowed you to do a number of 360 tours behind the scenes, so people could see parts of the environment you wouldn’t be able to see as a regular member of the public,” Cook says. Shopping for paraphernalia in virtual merch tents was part of the experience as well, which “proved to be quite popular,” he adds.
While Cook declines to discuss what’s next on Sinclair’s metaverse agenda, he makes clear there’s much more in store. “They’re very interested in looking at expanding beyond the Indian Wells event with other viewing engagements,” he says.
As for Deloitte’s ability to support more immersive, volumetric user experiences, those elements remain in the proof-of-concept stage with multiple clients kicking the tires on the possibilities. For example, Cook says, this “very leading-edge work” includes “two engagements right now on the next generation of AI headsets from a small company in Silicon Valley.”
For now Deloitte is providing “a great tool to supplement the television experience,” he adds. “I don’t think we’re at a stage quite yet where we’re going to replace TVs.”
But the great thing about what’s happening now with volumetric productions is that, as Cook puts it, “if you build it out in 3D, it’s relatively easy to pull it onto any device that could display the 3D experience, whether that’s the headsets, which still relatively few people own, or through your mobile phone or tablet device.” This was demonstrated with the USGA app at the U.S. Open, where on-site users could mix what they were watching on the course with immersive viewing of any other point in the tournament through one of the supported AR or mixed reality (MR) headsets.
Gerberman describes a similar stage of progress toward what he calls the “Holy Grail” of volumetric experiences that are on tap with six-degree-of-freedom (6 DoF) VR engagements with live sports. Whereas 3 DoF offered in VR sports viewing supported by NBC, the NBA, and others allows viewers to follow the action as if they were there from any of several fixed positions on the playing field, 6 DoF allows users to scope what’s happening in all directions as they virtually move around the arena.
The need to use volumetric technology to better engage with younger audiences is “one of the things we’re hearing from every sports league,” Gerberman says. “Kids are used to 3D experiences and navigating around 3D worlds,” he notes. “It’s just how kids interact with content. Volumetric technology is ripe for enabling those types of experiences with live sports.”
