An 180 page patent application from Magic Leap represents the first detailed depiction of how a virtual reality company believes we'll use their mind-bending hardware. Here, a toy creature leaps off a shelf. Magic Leap
An 180 page patent application from Magic Leap represents the first detailed depiction of how a virtual reality company believes we'll use their mind-bending hardware. Here, a toy creature leaps off a shelf.
Physically, Magic Leap will consist of a lightweight head-mounted device that will house a tiny projector comprised of bespoke prisms and lenses that will beam images onto the user’s retinas creating a "dynamic digitized light field signal." Magic Leap
Magic Leap believes the future of user interface will come in the form of ordinary objects imbued with virtual, interactive powers, like this app launching "key chain." Magic Leap
Magic Leap believes the future of user interface will come in the form of ordinary objects imbued with virtual, interactive powers, like this app launching "key chain."
Magic Leap's patent application describes how the system might work: "The totem may simply provide a physical surface, and the AR system may render a user interface so as to appear to a user to be on one or more surfaces of the totem. For instance, the AR system may render a virtual computer keyboard and virtual trackpad to appear on a surface of a thin rectangular plate of aluminum which serves as a totem." Magic Leap
Magic Leap's patent application describes how the system might work: "The totem may simply provide a physical surface, and the AR system may render a user interface so as to appear to a user to be on one or more surfaces of the totem. For instance, the AR system may render a virtual computer keyboard and virtual trackpad to appear on a surface of a thin rectangular plate of aluminum which serves as a totem."
Described as "Totems," these are tangible objects that combine physical craftsmanship and computational projection. Magic Leap
Described as "Totems," these are tangible objects that combine physical craftsmanship and computational projection.
Some of the embodiments are surprisingly low tech, like a cube where each face has a popular website overlaid on it. The user would switch between apps by rotating this cube. Magic Leap
Some of the embodiments are surprisingly low tech, like a cube where each face has a popular website overlaid on it. The user would switch between apps by rotating this cube.
Hand gestures, like those popularized by Leap Motion, are also viable interaction methods. Magic Leap
Hand gestures, like those popularized by Leap Motion, are also viable interaction methods.
A dozen illustrations show a guy in his basement interacting with a football game in novel ways. Magic Leap
A dozen illustrations show a guy in his basement interacting with a football game in novel ways.
Another illustration from the patent document shows a keyboard made from a soft rubber that would deform and provide haptic feedback while the head mounted display overlaid images of the characters on it in the user's eye. Magic Leap
Another illustration from the patent document shows a keyboard made from a soft rubber that would deform and provide haptic feedback while the head mounted display overlaid images of the characters on it in the user's eye.
Many of the concepts have a commercial focus. Magic Leap
Many of the concepts have a commercial focus.
The patent illustrates a series of motions that could turn any flat surface into an ad hoc workspace, Magic Leap
The patent illustrates a series of motions that could turn any flat surface into an ad hoc workspace,
Most of the drawings in the patent application make Magic Leap look like a fancy replacement for universal remote controls. Magic Leap
Most of the drawings in the patent application make Magic Leap look like a fancy replacement for universal remote controls.
A ghostly avatar of Toucan Sam would personally greet kids by name in the cereal aisle, rather than some off-brand spokesanimal. Magic Leap
A ghostly avatar of Toucan Sam would personally greet kids by name in the cereal aisle, rather than some off-brand spokesanimal.
A bracelet launches apps when a user caresses a charm emblazoned with the app's logo. Magic Leap
A bracelet launches apps when a user caresses a charm emblazoned with the app's logo.
Heads Up Displays (HUDs), long used by pilots, could be ported to automobiles. Magic Leap
Heads Up Displays (HUDs), long used by pilots, could be ported to automobiles.
From the patent application: "The user may, for example, perform a two finger gesture, for instance an expanding pinch gesture, dragging outward from a center point to locations where an upper left and a lower right corner of the virtual work portal or construct should be located." Magic Leap
From the patent application: "The user may, for example, perform a two finger gesture, for instance an expanding pinch gesture, dragging outward from a center point to locations where an upper left and a lower right corner of the virtual work portal or construct should be located."
When the "screen" is in your eye, TV sizes scale to the size of a room. Magic Leap
When the "screen" is in your eye, TV sizes scale to the size of a room.
Grocery shoppers would be treated to a virtual representation of Mario Batali, not some stock art chef. Magic Leap
Grocery shoppers would be treated to a virtual representation of Mario Batali, not some stock art chef.
With Magic Leap UI buttons are in the eye of the beholder. Magic Leap
With Magic Leap UI buttons are in the eye of the beholder.
The presence of the registered trademark "Sensorywear®" in the application, used to describe augmented reality devices, suggests that Magic Leap believes wearables will be a big part of the future of their product line. Magic Leap
The presence of the registered trademark "Sensorywear®" in the application, used to describe augmented reality devices, suggests that Magic Leap believes wearables will be a big part of the future of their product line.
There is the de rigueur drawing of augmented reality being used in surgery. Magic Leap
There is the de rigueur drawing of augmented reality being used in surgery.
Other hardware concepts seem to be pushing into the world of concept art, like a piece of hardware that's shaped like a lotus and presumably treats each petal as an input method. Magic Leap
Other hardware concepts seem to be pushing into the world of concept art, like a piece of hardware that's shaped like a lotus and presumably treats each petal as an input method.
When entire rooms become playgrounds for UI, the designer's imaginations can scale to fit the space. Magic Leap
When entire rooms become playgrounds for UI, the designer's imaginations can scale to fit the space.
Shopping lists will materialize in the consumer's eye. Magic Leap
Shopping lists will materialize in the consumer's eye.
The totems don't even need to by physical objects—even hands could work as totems in a pinch. Magic Leap
The totems don't even need to by physical objects—even hands could work as totems in a pinch.
Gamer's have no reason to fear, it appears that traditional game controllers will also be supported. Magic Leap
Gamer's have no reason to fear, it appears that traditional game controllers will also be supported.
One of the more interesting ideas: The concept of gamifying everyday labor, as seen with this road worker whose tasks take on elements of a racing simulator. Magic Leap
One of the more interesting ideas: The concept of gamifying everyday labor, as seen with this road worker whose tasks take on elements of a racing simulator.
Recuperating in a hospital could be made more pleasant by overlaying a tropical scene on a room. Magic Leap
Recuperating in a hospital could be made more pleasant by overlaying a tropical scene on a room.
Exercise bikes will be made more interactive by creating a virtual Tour de France. Magic Leap
Exercise bikes will be made more interactive by creating a virtual Tour de France.
Copious images of gloves, watches, and rings in the patent drawings back up that conjecture. Magic Leap
Copious images of gloves, watches, and rings in the patent drawings back up that conjecture.
Any surface or object could conceivably become a user interface. Magic Leap
Any surface or object could conceivably become a user interface.
An idea for an exercise class augmented with instructional overlays. Magic Leap
An idea for an exercise class augmented with instructional overlays.
A new patent application titled Planar Waveguide Apparatus with Diffraction Element(s) and System Employing Same sounds like a scientific snoozefest, but just also might provide a playbook for the next decade of interaction design.
The surprisingly broad patent application was filed by Magic Leap, the secretive, Florida-based “Cinematic Reality” startup that recently received $542 million dollars of venture capital from Google, Legendary Entertainment, and Andresseen Horowitz. And its 180 pages represent the first detailed depiction of how the augmented-reality company believes we’ll use this mind-bending hardware.
Magic Leap has been secretive about how their system works technically, but a plethora of disclosures in their filings provide the broad outline. A lightweight head-mounted device will house a tiny projector comprised of bespoke prisms and lenses that will beam images onto the user’s retinas creating a “dynamic digitized light field signal.” Apps, powered by mobile devices or body worn computers, will generate a steady stream of fantastic creatures and surreal tableaus delivered with stereo speakers and at 60 frames per second. Infrared positioning cameras, GPS modules, and multi-axis accelerometers will blend these otherworldly images with more banal surroundings of a basement man cave.
The big question is how we’ll interact with devices like this. Sitting in front of a keyboard defeats the purpose. Voice commands like those used with Google Glass are clunky, poorly suited for public use, and inefficient. Even Oculus, the company that captured the hearts and eyes of VR fanatics, has said that they may not include native input support for the first few generations of their system.
What’s the Future of UI? Just One Word: Totems
Magic Leap believes the future of user interface will come in the form of ordinary objects imbued with virtual, interactive powers. Described as “Totems,” these are tangible objects that combine physical craftsmanship and computational projection. Virtual worlds and physical objects are melded in the wearer’s retina. Put another way, with Magic Leap UI buttons are in the eye of the beholder.
The simplest embodiment of this concept is a “computer mouse,” which could actually be any block of plastic, wood, metal, or glass. Rather than relying on sensors built into the device, front-facing cameras on the Magic Leap headset could theoretically track any piece of material that’s been defined as a “mouse.” The Magic Leap headset would constantly be plotting it’s position, direction, distance, speed, and acceleration while feeding those cues back to the augmented reality display.
Another illustration from the patent document shows a keyboard made from a soft rubber that would deform and provide haptic feedback while the head mounted display overlaid images of the characters on it in the user’s eye.
Some of the totem concepts are more elegant. Illustrations show wearers using the popular pinch to zoom gesture to create a “screen” on an odd-sized piece of metal. The concept is described in detail here: “The totem may simply provide a physical surface, and the AR system may render a user interface so as to appear to a user to be on one or more surfaces of the totem. For instance, the AR system may render a virtual computer keyboard and virtual trackpad to appear on a surface of a thin rectangular plate of aluminum which serves as a totem.”
Some of the embodiments are surprisingly low tech, like a cube where each face has a popular website overlaid on it. The user would switch between apps by rotating this cube. A bracelet offers similar app switching behavior, but requires the user to caress a charm emblazoned with the app’s logo.
The presence of the registered trademark “Sensorywear®” in the application, used to describe augmented reality devices, suggests that Magic Leap believes wearables will be a big part of the future of their product line. Copious images of gloves, watches, and rings in the patent drawings back up that conjecture.
In addition to these forward-looking totems there will also be traditional gesture-based controls, like those popularized by the Leap Motion sensor and Microsoft Kinect. The patent illustrates a series of motions that could turn any flat surface into an ad hoc workspace, described here: “The user may, for example, perform a two finger gesture, for instance an expanding pinch gesture, dragging outward from a center point to locations where an upper left and a lower right corner of the virtual work portal or construct should be located.”
Gamer’s have no reason to fear, it appears that traditional game controllers will also be supported. Other hardware concepts seem to be pushing into the world of concept art, like a piece of hardware that’s shaped like a lotus and presumably treats each petal as an input method.
Otherworldly Tech, Yawn-Worthy Use Cases
There’s a stark difference between the applications depicted on the Magic Leap website and in their patent filing. Illustration: Magic Leap
The most surprising thing about the Magic Leap patent is how pedestrian most the applications for this breakthrough technology appear to be.
On the Magic Leap website, the uses cases are presented as equal parts Silicon Valley and Salvador Dalí. Whales list lazily over Miami Beach while dragons populate the sky. Most of the drawings in the patent application make Magic Leap look like a fancy replacement for universal remote controls.
A dozen illustrations show a guy in his basement interacting with a football game in novel ways. There is the de rigueur drawing of augmented reality being used in surgery. Exercise bikes will be made more interactive by creating a virtual Tour de France. Physical desktops will take on the interactive properties of their computation counterparts. These aren’t bad applications, just similar to what was promised by Google Glass.
One of the more interesting aspects of the application is the surprising specificity and highly branded nature of the drawings. Grocery shoppers would be treated to a virtual representation of Mario Batali, not some stock art chef. A ghostly avatar of Toucan Sam would personally greet kids by name in the cereal aisle, rather than some off-brand spokesanimal. Twitter and Facebook logos are emblazoned on physical icons. This makes the drawings easier to parse, but also seems to play into the technical implementation of the system.
The patent calls out computer vision as a key component of many of these UI concepts and aims to solve the tricky challenges it presents using “Object Recognizers.” These would be chunks of code that make it easy for Magic Leap software to identify a specific trademark, like the Starbucks’ logo, in a crowded field of view. This has the side benefit of making the tool work better while also potentially strengthening the company’s business model.
Many have posited uses for AR and VR, but few have illustrated them with the verve and detail that Magic Leap has. The breadth and detail of the patent’s claims make the system seem like science fiction, but Magic Leap founder Rony Abovitz previously developed FDA-cleared has previously developed FDA-cleared surgical robots that he likened to the Droids from Star Wars. He sold that company, with a similarly futuristic premise, for $1.65 billion dollars, so you might say the force is strong with this one.
No comments:
Post a Comment