His group decided to find out. They built the new, diversified version of AlphaZero, which includes multiple AI systems that trained independently and on a variety of situations. The algorithm that governs the overall system acts as a kind of virtual matchmaker, Zahavy said: one designed to identify which agent has the best chance of succeeding when it’s time to make a move. He and his colleagues also coded in a “diversity bonus”—a reward for the system whenever it pulled strategies from a large selection of choices.
When the new system was set loose to play its own games, the team observed a lot of variety. The diversified AI player experimented with new, effective openings and novel—but sound—decisions about specific strategies, such as when and where to castle. In most matches, it defeated the original AlphaZero. The team also found that the diversified version could solve twice as many challenge puzzles as the original and could solve more than half of the total catalog of Penrose puzzles.
“The idea is that instead of finding one solution, or one single policy, that would beat any player, here [it uses] the idea of creative diversity,” Cully said.
With access to more and different played games, Zahavy said, the diversified AlphaZero had more options for sticky situations when they arose. “If you can control the kind of games that it sees, you basically control how it will generalize,” he said. Those weird intrinsic rewards (and their associated moves) could become strengths for diverse behaviors. Then the system could learn to assess and value the disparate approaches and see when they were most successful. “We found that this group of agents can actually come to an agreement on these positions.”
And, crucially, the implications extend beyond chess.
Real-Life Creativity
Cully said a diversified approach can help any AI system, not just those based on reinforcement learning. He has long used diversity to train physical systems, including a six-legged robot that was allowed to explore various kinds of movement, before he intentionally “injured” it, allowing it to continue moving using some of the techniques it had developed before. “We were just trying to find solutions that were different from all previous solutions we have found so far.” Recently, he has also been collaborating with researchers to use diversity to identify promising new drug candidates and develop effective stock-trading strategies.
“The goal is to generate a large collection of potentially thousands of different solutions, where every solution is very different from the next,” Cully said. So—just as the diversified chess player learned to do—for every type of problem, the overall system could choose the best possible solution. Zahavy’s AI system, he said, clearly shows how “searching for diverse strategies helps to think outside the box and find solutions.”
Zahavy suspects that in order for AI systems to think creatively, researchers simply have to get them to consider more options. That hypothesis suggests a curious connection between humans and machines: Maybe intelligence is just a matter of computational power. For an AI system, maybe creativity boils down to the ability to consider and select from a large enough buffet of options. As the system gains rewards for selecting a variety of optimal strategies, this kind of creative problem-solving gets reinforced and strengthened. Ultimately, in theory, it could emulate any kind of problem-solving strategy recognized as a creative one in humans. Creativity would become a computational problem.
Liemhetcharat noted that a diversified AI system is unlikely to completely resolve the broader generalization problem in machine learning. But it’s a step in the right direction. “It’s mitigating one of the shortcomings,” she said.
More practically, Zahavy’s results resonate with recent efforts that show how cooperation can lead to better performance on hard tasks among humans. Most of the hits on the Billboard 100 list were written by teams of songwriters, for example, not individuals. And there’s still room for improvement. The diverse approach is currently computationally expensive, since it must consider so many more possibilities than a typical system. Zahavy is also not convinced that even the diversified AlphaZero captures the entire spectrum of possibilities.
“I still [think] there is room to find different solutions,” he said. “It’s not clear to me that given all the data in the world, there is [only] one answer to every question.”
Original story reprinted with permission from Quanta Magazine, an editorially independent publication of the Simons Foundation whose mission is to enhance public understanding of science by covering research developments and trends in mathematics and the physical and life sciences.