Art-ificial Intelligence? Algorithm Sorts Paintings Like a Person

Featured on livescience.com

From assembly-line work to self-driving cars , computers are taking over many tasks once performed by humans. Artistic jobs, however, have been relatively safe — until now.

A team of researchers has developed an artificial intelligence(AI) program that can classify famous works of art based on their style, genre or artist  — tasks that normally require a professional art historian.

The AI program classified approximately 80,000 works of art with unprecedented accuracy, and revealed surprising connections among different artists  and painting styles, stated the study, which was posted to the preprint server arXiv on May 5.

"We're definitely not replacing art historians, but with a growing number of paintings in online collections, we need an automatic tool" for organizing them, said study researcher Babak Saleh, a computer scientist at Rutgers University in New Brunswick, New Jersey.

The field of computer vision has advanced significantly in recent years, but AI still lags far behind humans in basic tasks. A human can look  at a painting and easily draw inferences from it, such as whether it's a portrait or a landscape, whether the style is impressionist or abstract, or who the artist was.

"The average person can tell these things, but that's very challenging when it comes to a machine," said study researcher Ahmed Elgammal, who is also a computer scientist at Rutgers. "Our goal is to push what machine intelligence can do."

To create a machine capable of classifying art, Saleh and Elgammal used a database of more than 80,000 paintings by more than a 1,000 artists across 15 centuries, spanning 27 different styles.

The researchers used a variety of machine-learning algorithms to pick out particular features in a subset of the paintings, including low-level attributes, such as colors and edges, as well as more abstract ones, such as what an object is — whether it's a horse or a human, for example. One approach they used is known as deep learning, a method employed by Google and other companies in image searches and translation tools.

Then, the researchers tested their algorithm on a set of paintings the machine had never seen, and it performed remarkably well. The program was 63 percent accurate at identifying the artist, about 60 percent accurate at figuring out the genre and about 45 percent accurate at determining the style.

It's difficult to compare the AI's performance to that of an art historian, because the historian has a lot of prior knowledge, Elgammal said. However, he estimated the algorithms would "do much better than the average human," though "not as good as an expert."

In addition, the paintings the algorithm had trouble categorizing offeredinsight  into the influences different painters may have had on each other. For example, the algorithm had difficulty distinguishing between a painting by the 18th-century Danish painter Christoffer Wilhelm Eckersberg in the neoclassical style and one by the early 19th-century Dutch painter Cornelis Vreedenburgh in the impressionist style.

These parallels are no surprise to art historians, but are nevertheless impressive for a computer program, the researchers said.

The study has not yet been published in a peer-reviewed journal.