Algorithms that "Don't See Color"

Today, algorithmic models are shaping important decisions in domains such as credit, employment, or criminal justice. At the same time, these algorithms have been shown to have discriminatory effects. Some organizations have tried to mitigate these effects by removing demographic features from an algorithm's inputs. If an algorithm is not provided with a feature, one might think, then its outputs should not discriminate with respect to that feature. This may not be true, however, when there are other correlated features.

In this paper, we explore the limits of this approach using a unique opportunity created by a lawsuit settlement concerning discrimination on Facebook's advertising platform. Facebook agreed to modify its Lookalike Audiences tool - which creates target sets of users for ads by identifying users who share "common qualities" with users in a source audience provided by an advertiser - by removing certain demographic features as inputs to its algorithm. The modified tool, Special Ad Audiences, is intended to reduce the potential for discrimination in target audiences. We create a series of Lookalike and Special Ad audiences based on biased source audiences - i.e., source audiences that have known skew along the lines of gender, age, race, and political leanings. We show that the resulting Lookalike and Special Ad audiences both reflect these biases, despite the fact that Special Ad Audiences algorithm is not provided with the features along which our source audiences are skewed. More broadly, we provide experimental proof that removing demographic features from a real-world algorithmic system's inputs can fail to prevent biased outputs. Organizations using algorithms to mediate access to life opportunities should consider other approaches to mitigating discriminatory effects.