When we sit down to solve a jigsaw puzzle, there’s always one thing we take for granted: the picture on the box. Without that point of reference, we’d be pulling our hair out, trying and failing to rebuild a jumbled pile of miscellaneous pieces.
That’s exactly what was happening in the ancient Roman city of Pompeii, where over 10,000 fragmented pieces from 2,000-year-old frescoes have been lying around for decades, waiting for someone to solve the puzzle. Now, a team of scientists led by the Venice-based Italian Institute of Technology may have found a solution: train a robot to do it.
A key part of the project will be teaching the algorithm how to study like an archeologist and think like a puzzle master. The puzzle-solving AI was developed in collaboration with a team at Ben Gurion University of the Negev, in Israel, and it works like an infinitely more intricate version of the popular memory game “find the pair.” The computer software compares all the fragments in pairs and rates their degree of similarity based on the shape of the pieces, how they fit together, plus how the illustrations on the fragments match up. Normally, this process can be done manually (using a computer), but the team is now teaching the algorithm how to compare pieces on its own.
Thanks to a team of archeologists from the University of Lausanne, who have attempted to solve the puzzle in the past, they already have about a dozen reconstructed clusters (about 10 pieces each) that they are feeding the algorithm. If the computer can put those pieces back together, they will know the system can be deployed at a larger scale.
The robot will be deployed in Pompeii sometime next summer, but for now, the scientists are working on several projects in parallel. While one team is building the algorithm, another is 3D scanning a large sample of fragments so they can be put into the database (once the robot is fully operational, it will scan them on its own). And another team is working on the physical infrastructure and robotic hands that will eventually pick up the pieces and rebuild the frescoes. “One day, you will take all the pieces, put them into a room, lock the door, come back after a few days, and you will find the fresco completely reassembled,” says Marcello Pelillo, a professor of computer science and artificial intelligence at the University of Venice. (Though he admits things probably won’t go quite so smoothly.)The fragments come from two separate rooms (including the ceilings) in a building called the House of Painters at Work, (named because artists were in the middle of painting when Mount Vesuvius erupted in A.D. 79.) They run the gamut from tiny fragments to palm-sized pieces, most of them damaged, and many of them missing. The scientists are working with archeologists and art historians, who will narrow down the pieces to an initial 1,000 they think belong in the same cluster, or at least on the same wall.
Within a year or two, Pelillo says they should have a preliminary image of the frescoes, but a complete and final image will take longer. When the robotic platform is finished, the plan is for the robot to do it all at once: “The robot will do the scanning itself, then after doing the scanning, it will solve the puzzle, then after solving the puzzle it will reassemble it,” says Pelillo. (The set-up will look like a bridge, with two robotic arms suspended from a metal frame.)
At the end of the project, the frescoes will be displayed in the Archeological Park of Pompeii for visitors to experience, but they will inform further research, too. Ariana Traviglia, the director of the IIT Center for Cultural Heritage and Technology, says we can learn a lot from the reconstructed frescoes and the patterns on them.
Eventually, the team could put all of this into practice in other cultural heritage sites, like the host of ancient churches that crumbled during the earthquakes in Central Italy four years ago. “All over the world, we have so many pieces of broken frescoes and broken objects from our past, and they are so tiny that we don’t have the time and personnel to put them together,” she says. “It will really change the things we can do.”