Visual Culture

An online image database will remove 600,000 pictures after an art project revealed the system’s racist bias.

Christy Kuesel
Sep 24, 2019 4:20PM, via The Art Newspaper

Exhibition view of “Training Humans” by Kate Crawford and Trevor Paglen, Osservatorio Fondazione Prada. Photo by Marco Cappelletti. Courtesy Fondazione Prada.

ImageNet, a popular online database of images, will remove 600,000 pictures of people from its system after an art project revealed the depths of the racial biases of the system’s artificial intelligence.

ImageNet Roulette, an initiative by artist Trevor Paglen and AI researcher Kate Crawford, allows users to upload photos of themselves, which are then classified using ImageNet technology. The site launched last week as part of an exhibition at the Fondazione Prada in Milan titled “Training Humans.” Sometimes, the results were funny—you might be classified a “pipe smoker” or a “microeconomist.” But other results revealed the inherent bias in the system—a woman might be labeled as a “slut,” while African American users reported being labeled as a “wrongdoer” or with a racial slur.

“This exhibition shows how these images are part of a long tradition of capturing people’s images without their consent, in order to classify, segment, and often stereotype them in ways that evoke colonial projects of the past,” Paglen told The Art Newspaper.

Part of the issue may stem from how images on ImageNet were originally categorized. Researchers at Princeton and Stanford created the database in 2009 by collecting photos from websites like Flickr. Workers at Amazon Mechanical Turk, a marketplace that allows businesses to outsource labor for extremely low wages, then categorized the photos. By defining photos with terms ranging from “cheerleader” to “failure, loser, non-starter, unsuccessful person,” misogynistic, racial, and other biases were formalized in the system. ImageNet was frequently used to train AI systems and judge their accuracy, meaning the implications of these categorizations go far beyond the database itself.

In its announcement about removing about half of the images of people from its site, ImageNet did not reference the ImageNet Roulette project. But the timing of the statement—five days after Pagen and Crawford’s exhibit opened—suggests the removal is related. The database says it has identified 438 “unsafe” and “offensive” categories of people, while another 1,155 labels are “sensitive,” or potentially offensive. Any image related to one of these categories will be removed.

As for the ImageNet Roulette project, Paglen and Crawford said they will take the site down after this Friday, though it will remain at the physical art installation until February 2020. On the website, the duo wrote: “ImageNet Roulette has made its point—it has inspired a long-overdue public conversation about the politics of training data, and we hope it acts as a call to action for the AI community to contend with the potential harms of classifying people.”

Christy Kuesel