A leading-edge research firm focused on digital transformation.
Subscriber Account active since
Facebook’s vision for a metaverse includes an array of body-worn sensors and devices that track and predict movement to create a hyperrealistic digital world with little privacy or user safety built in, according to an Insider review of hundreds of patents recently granted to the social media giant.
Facebook CEO Mark Zuckerberg says this interactive realm will take at least a decade to scale, but the company’s engineers have already been busy designing the technical ingredients — and patenting their inventions.
“Delivering a sense of presence, like you’re right there with another person, that’s the holy grail of online social experiences,” Zuckerberg told analysts late last year.
He’s betting the future of Facebook on this technology, and is investing $10 billion a year to make it happen, even changing the company’s name to Meta to highlight his new priority. To succeed, though, the experience must be so compelling that billions of people give up their smartphones and rely instead on headsets and other wearable devices to access the platform.
Patents filed over last year or so, and granted in the last four months, reveal a company hastily inventing the software, hardware and other infrastructure that will support the metaverse. Filings like these often never result in end products or services, but they’re a useful guide to tech companies’ future plans. A Facebook spokesperson declined to comment on why it filed these patents.
What’s clear, though, is that Facebook is trying to build a 3D environment that is much more realistic than the current iteration, where sharp-edged humanoid avatars move awkwardly. The company’s future metaverse will be populated with avatars capable of authentic eye and body movement, expressions, poses, interactions and sensations, with clothing that wrinkles with movement. And these avatars will be in a virtual world where objects can be realistically grabbed, moved, and altered, in surroundings that can be recreated from an environment anywhere in the world. All while you sit in your bedroom, possibly feeling as though you are in and seeing a vast expanse.
Facebook has new patents covering all of this and more. There’s one for a “suspend mode” where you can freeze an AR environment and carry out a separate task, and one for “collision avoidance” in an artificial reality system to keep people from running into the edge of a table, for instance. One for “comfortable navigation in large-scale virtual worlds when constrained by small-scale physical environments” through the quick creation of new virtual paths. Another for “generating accurate and realistic clothing” requires body mapping. There’s one for “avatar fidelity and personalization” that can ostensibly have a person’s avatar be a recreation of their individual physical features, instead of the current cartoon-like approximation.
Another patent for “gesture-based casting and manipulation of virtual content” will allow an avatar to seamlessly throw, grab, pinch and otherwise interact with a virtual object. Facebook wants these virtual objects to be rendered in real time and with the change in your eye gaze, whether they are copies from reality or creations, according to several patents covering their representation.
This push toward realism is the central goal for companies looking to build 3D worlds, according to Jeremy Bailenson, founder of Stanford University’s Virtual Human Interaction Lab. He’s been working with VR for 20 years and also cofounded Strivr, a company that does large scale trainings through VR.
“Every company that’s building any AR/VR technology is trying to build photorealistic avatars that can interact and that also have low latency and are rendered quickly,” Bailenson said.
Not one company is there yet, he noted, nor has one been able to build a system that can host even 10 or 15 photorealistic avatars at once and have them interact. That will take a decade or more, Bailenson estimated. “There will be a pace this stuff grows at, and it will be slower than Silicon Valley would like,” he said.
Beyond the lack of realism, Bailenson came across another issue last year, when he taught a class on augmented reality and virtual reality which took place partially in a metaverse for several weeks.
“It sounds ridiculous, but what’s lacking the most is something to do,” he said. “If my class hadn’t been about VR, it would have been a real challenge to host a class there.” He also has a 30 minute limit for wearing VR goggles, mainly because, despite some improvements, they’re uncomfortable to wear for long periods of time.
In order for a metaverse to go more mainstream, “we need one or both of these things to happen: the hardware gets better or the experiences are so incredible you don’t mind wearing them,” Bailenson said.
A couple of Facebook’s patents focus on improving the range and level of experiences in a metaverse. One details technology for generating “spectator images,” or a first-person perspective that one user can share with another person. That could possibly be used to invite friends to watch a concert live, even though they’re not physically at the event. Another patent that describes the “porting of physical objects into virtual reality” shows the use of a real computer while remaining in an AR/VR environment. There’s another for various “notification triggers” that pop up while you’re in a 3D world to be accessed or dismissed by simply looking at them.
The typical approach to VR and AR hardware isn’t going away any time soon, as nearly all of the relevant patents contemplate using the metaverse via head-mounted display. Some images in the filings show a modified version of a headset that is closer to glasses than goggles, and even a headband.
Insider’s review of Facebook’s patents found essentially no technology thus far related to privacy or user safety, despite Zuckerberg and other executives insisting that the metaverse is being built with such things in mind. There are, however, patents covering the delivery of personalized advertising and content while consumers are in this new digital realm.
To have a realistic metaverse, the amount of personal data required will likely be immense, even compared to the mountain of information Facebook already collects on the billions of people who use its social networks and messaging services.
“We think these companies have data access now — no,” Jeanine Turner, professor of communication, culture, and technology at Georgetown University, said around the time Facebook announced its metaverise pivot. “It’s mind blowing what they will have.”
Several patents granted to Facebook in recent months cover the minute tracking of eye, face and body movements through additional “wearables.” There’s an apparatus for “acoustic sensing” that would allow for the creation of sound from virtual objects when they are touched, a “magnetic sensor system” to be worn around the torso for “body pose tracking,” and a system, for gloves and other devices, that simulates touch.
There are also a number of apparent updates to waveguide technology, core to AR and VR capabilities, that more extensively track eye and facial movements. This is needed for “pupil steering,” the movement of an avatar’s eyes, and its facial expressions. Then there’s a group of sensors to be worn all over the body for “predicting” your “musculo-skeletal position,” or the movement of your entire body and its various bones, muscles and ligaments.
All of these things would likely log huge amounts of data about the wearer. “With Facebook, your data is their product,” said Owen Vaughan, the director of research at nChain, a company focused on data security and privacy. “Creating a metaverse opens up a lot more risk in terms of privacy and security.”
He added that now, when foundational uses and technology for a metaverse are being created, would be the time for privacy and safety guardrails to be patented and applied. “It may be impossible to integrate afterward,” he added.
“It’s very concerning that security and privacy is not there in the patents,” Vaughan said. “It should be, and it could be.”
A leading-edge research firm focused on digital transformation.