For years, curators have attempted to use the latest technology to help modern audiences connect with age-old art – and the paintings of Dutch artist Rembrandt van Rijn are at the forefront. Some curators have even used apps, much like the Snapchat filters found on your smartphone, to make naturalistic masterpieces from the Dutch Golden Age even more realistic.
“Our paintings come to life at night! the official Instagram account of the Rijksmuseum Amsterdam, which houses most of Rembrandt’s works, joked in a message showing this trick in play. 17th century portraits often already looked viewers in the eye, but now, thanks to deepfake technologies, they blink and move too. It is disturbing to say the least.
Although these deepfakes to make enticing marketing campaigns, they are not really revolutionary from a scientific point of view. If we want to deepen the relationship between art and artificial intelligence, we need to look to other more elaborate research projects that have used Rembrandt as a personal test subject.
In 2014, a team of data scientists from Microsoft joined forces with AI and 3d printing experts to build a computer program capable of creating a Rembrandt-style image in the painter’s unrivaled style. Nicknamed “The next Rembrandt“, the project began when a convolutional neural network (a type of artificial neural network used in image recognition) gained access to a massive 150 GB database containing high-resolution graphics of Rembrandt’s work .
By sifting through the course load, the network learned to recognize combinations of shapes and colors for the real-life objects they represented. The program analyzed 67 common features in paintings – including the nose, hands, and eyes – and then mixed various images to 3D print what the researchers called an “original” Rembrandt.
A painting printed in 3D using data from Rembrandt’s works. (Credit: ING Group / Flickr)
Whether the portrait can truly be considered original has become a matter of debate. It was, after all, neither created from the imagination nor taken from life, but distilled from the encyclopedic work of Rembrandt. “How can a computer reproduce the humanity of Rembrandt’s portrait of his lover Hendrickje Stoffels?” »Art critic Jonathan Jones opposes it The Guardian. “He would have to sleep with her first.”
While the next Rembrandt wasn’t the milestone in creative AI his team had hoped for, the project helped us see Rembrandt from a different perspective. Detailed 3D scans of the painter’s canvases revealed where he applied the most pressure to his brush. The skin generally had a smooth and subtle texture, while Rembrandt tended to apply rough, emotional lines to clothing, especially later in his career.
More important, perhaps, the two-year project helped lay the groundwork for another more recent attempt to emulate Rembrandt’s genius using the power of AI. In 2019, the Rijksmuseum launched a multi-million dollar project restoration mission called Operation Night Watch. Earlier this year, restorers used computer technology to recreate sections of the eponymous painting that were lost centuries ago.
Convert a copy to an original
In the 17th century, the politicians in charge of Amsterdam’s town hall came into possession of the Night watch. When it didn’t fit on the wall chosen by its new owners, they cut off all four edges of the artist’s most famous painting.
These cosmetic adjustments completely changed its overall effect. The absence of two figures on the far left of the painting pushed Captain Frans Banninck Cocq sideways towards the center, upsetting Rembrandt’s composition. Among the casualties was also an elegant balustrade capturing the shadow of the aforementioned captain, which would have given his otherwise static squadron a sense of forward momentum.
Visitors see Rembrandt’s The Night Watch at the Rijksmuseum in 2016 (Photo credit: InnaFelker / Shutterstock)
Wishing visitors could discover Night watch as Rembrandt had originally planned, the Rijksmuseum enlisted its principal scientist, Robert Erdmann, to restore these long-lost sections using a neural network capable of doing what human hands cannot: transform a painted copy of Rembrandt’s original masterpiece into something that looks as close as possible to the real deal.
While The Next Rembrandt researchers could reuse old paints, Erdmann had to settle for an imitation of Night watch. Gerrit Lundens, a contemporary of Rembrandt, copied the original painting before its outer edges are removed. While Lundens faithfully reproduced every little detail, his art style was so different from Rembrandt’s that Erdmann couldn’t just copy that version over the original.
After adjusting the perspectives of the two paintings to make sure they overlapped perfectly, Erdmann asked a neural network to translate random tiles from Lundens’ copy into Rembrandt’s style. He then noted each of the network’s attempts until its release could fool even an art critic. At this point, Erdmann asked the AI to repaint the outer edges of Lundens. Night watch, which he then stuck on that of Rembrandt.
Different methods of machine learning
While the AI used for The Next Rembrandt project improved by tracking the similarities between multiple works, Erdmann’s network learned by comparing the works of two separate artists. By noticing what Rembrandt did differently from Lundens, the network deduced what made each painter’s style unique. He then applied this knowledge to the missing sections present in the copy.
By studying Rembrandt through the prism of a computer program, humans can perceive things that we might have missed with the naked eye. When “translating” the Lundens copy, for example, the network replaced the copyist’s saturated palace with darker, duller colors. He also accentuated the contrast between lights and shadows. In the art world, this technique is known as “chiaroscuro”, and it was one of Rembrandt’s favorite tricks.
Practicing chiaroscuro is one thing, but understanding what it means is another. Does Erdmann’s AI capture the aesthetic quality of technique the same way an artist like Rembrandt did? Probably not. A more likely explanation is that the software found that strengthening the contrasts between light and dark resulted in higher ratings received from Erdmann, which is why it continued to do so.
After the AI translated the missing sections of the print in Rembrandt style, they were printed on canvas and coated with a gloss similar to the original. Instead of sewing the work of man and machine together, the missing sections were attached to plates placed less than an inch in front of the original Night watch, without really touching it.
Erdmann said that growing up, he greatly admired the polymath Leonardo da Vinci for his ability to make breakthroughs in several academic disciplines. When asked to join Operation Night Watch, the programmer saw a rare opportunity to do the same, combining his knowledge of coding with his love for art.