ArtEmis: Affective language for visual art

March 7, 2021, KAUST, Saudi Arabia – KAUST Assistant Professor of Computer Science Mohamed Elhoseiny has developed, in collaboration with Stanford University, CA, and École Polytechnique (LIX), France, a large-scale dataset to train AI to reproduce human emotions when presented with artwork.

The resulting paper, “ArtEmis: Affective Language for Visual Art,” will be presented at the Conference on Computer Vision and Pattern Recognition (CVPR), the premier annual computer science conference, which will be held June 19-25, 2021.

Described as the “Affective Language for Visual Art,” ArtEmis’s user interface has seven emotional descriptions on average for each image, bringing the total count to over 439K emotional-explained attributions from humans on 81K pieces of art from WikiArt.

“Before this project, most machine learning models were based on factual description datasets,” Elhoseiny explains. “For example, with ‘a bird is perched on the chair,’ Artemis expanded on the image description by requesting that people also add the emotions they felt when observing the artwork, which incorporated complex metaphoric language and abstract ideas,” he adds.

The initial design was inspired by Northeastern University’s, U.S., Distinguished Professor of Psychology Lisa Feldman Barrett, and is explained in her book “How Emotions Are Made: The Secret Life of the Brain.” In her book, Barrett showed how stereotypical faces helped improve people’s identification of constructed emotions. “We intentionally used emojis in our interface because Barrett’s experiments proved that recognizing emotions is a challenging problem, even for human beings.”, Elhoseiny adds. Data generated by ArtEmis enable the building of AI systems beyond the classical view of emotions that are currently adopted in affective AI industrial products based on facial expression recognition. Affective image description models based on ArtEmis-like data may help people to have a more positive experience by connecting better to artworks and appreciating them. In line with Barret’s view, this may also open the door to using affective AI to alleviate mental health problems.

The researchers then carried out human studies to show the unique features of the ArtEmis dataset. For example, ArtEmis requires more emotional and cognitive maturity compared with well-established vision and language datasets. The research was also validated via a user study where participants were asked whether the descriptions were pertinent to the associated artwork.

“But we did not stop there. To show the potential of affective neural speakers, we also trained image captioning models in both grounded and nongrounded versions on our ArtEmis dataset. The Turing Test showed that generated descriptions closely resemble human ones,” says Elhoseiny.

ArtEmis started while Dr. Elhoseiny was a visiting professor at Stanford University with Prof. Guibas. In collaboration with Stanford’s Paul Pigott, professor of computer science and one of the leading authorities in Computer vision and Graphics, Elhoseiny co-build a large-scale art and language dataset as a partnership project with Panos Achlioptas, a Stanford Ph.D. student of Prof. Guibas, who adopted the proposal and made significant efforts in making this project a solid reality. The project implementation was also supported by Kilich Hydarov, an M.S./Ph.D. candidate from the KAUST Vision-CAIR group. The collaboration also benefited from the expertise of LIX Ecole Polytechnique’s Maks Ovsjanikov, professor of computer Science and one of the leading graphics and vision researchers.

“Our dataset is novel as it concerns an underexplored problem in computer vision: the formation of emo-linguistic explanations grounded on visuals. Specifically, ArtEmis exposes moods, feelings, personal attitudes and abstract concepts, such as freedom or love, induced by a wide range of complex visual stimuli,” concludes Elhoseiny.

The dataset can be accessed at

https:/

.

King Abdullah University of Science and Technology (KAUST) advances science and technology through distinctive and collaborative research integrated with graduate education. Located on the Red Sea coast in Saudi Arabia, KAUST conducts curiosity-driven and goal-oriented research to address global challenges related to food, water, energy, and the environment.

Established in 2009, KAUST is a catalyst for innovation, economic development and social prosperity in Saudi Arabia and the world. The University currently educates and trains master’s and doctoral students, supported by an academic community of faculty members, postdoctoral fellows and research scientists.

With over 100 nationalities working and living at KAUST, the University brings together people and ideas from all over the world.

Visit kaust.edu.sa

###