Google AI Plays Atari Like the Pros

Google DeepMind

Last year Google shelled out an estimated $400 million for a little-known artificial intelligence company called DeepMind. Since then, the company has been pretty tight-lipped about what’s been going on behind DeepMind’s closed doors, but here’s one thing we know for sure: There’s a professional videogame tester who’s pitted himself against DeepMind’s AI software in a kind of digital battle royale.

The battlefield was classic videogames. And according to new research published today in the science magazine Nature, Google’s software did pretty well, smoking its human competitor in a range of Atari 2600 games like Breakout, Video Pinball, and Space Invaders and playing at pretty close to the human’s level most of the time.

Google didn’t spend hundreds of millions of dollars because it’s expecting an Atari revival, but this new research does offer a hint as to what Google hopes to achieve with DeepMind. The DeepMind software uses two AI techniques—one called deep learning; and the other, deep reinforcement learning. Deep-learning techniques are already widely used at Google, and also at companies such as Facebook and Microsoft. They help with perception—helping Android understand what you’re saying, and Facebook know who’s photo you just uploaded. But until now, nobody has really matched Google’s success at merging deep learning with reinforcement learning—those are algorithms that make the software improve over time, using a system of rewards.

By merging these two techniques, Google has built a “a general-learning algorithm that should be applicable to many other tasks,” says Koray Kavukcuoglu, a Google researcher. The DeepMind team says they’re still scoping out the possibilities, but clearly improved search and smartphone apps are on the radar.

But there are other interesting areas as well. Google engineering guru Jeff Dean says that AI techniques being explored by Google—and other companies—could ultimately benefit the kinds of technologies that are being incubated in the Google X research labs. “There are potential application in robots and self-driving-car kinds of things,” he says. “Those are all things where computer vision is pretty important.”

Google says that its AI software, which it’s dubbed the “Deep Q network agent,” got 75 percent of the score of its professional tester in 29 of the 49 games it tried out. It did best in Video Pinball.

Deep Q works best when it lives in the moment—bouncing balls in Break Out, or trading blows in video boxing—but it doesn’t do so well when it needs to plan things out in the long-term: climbing down ladders and then jumping skeletons in order to retrieve keys in Montezuma’s Revenge, for example. Poor old Deep Q scored a big fat zero in that game.