A Story Of Discrimination And Unfairness

Aylin Caliskan-Islam, Joanna Bryson, and Arvind Narayanan

Bias can be present individually in humans or collectively in corporations, groups, or cultures. We observe collective and individual implicit bias through analyzing writing in an automated way. Automating bias observations is possible through incorporating machine learning and natural language processing techniques to text analysis. The use of such an automated method makes large scale analysis possible with a variety of settings to compare and contrast bias in di erent conditions, such as subject of interest, time, location, culture, and language. Our proposed approach is a step towards a principled method for quantifying bias and fairness in language models that are used digital communications.

Machine learning algorithms and models have been criticized for incorporating bias from the data they train on. Eliminating bias in machine learning has been limited to controlling parameters at the algorithmic level to avoid overfitting, which does not prevent implicit bias from getting embedded to the model and revealing itself at the contextual level. Based on this knowledge, we train language models on writings of subjects of interest to generate a semantic space represented with word embeddings. Each word embedding quanti es a word used by the subject as a vector, where the dimensions of the vector represent a combination of contexts. We focus on the numeric vectors of concepts that have been used in bias studies in the literature, such as gender, racism, religion, and age. Then, we measure associations between concepts and potentially biased terms to observe implicit bias through spatial relations.

We investigate bias in famous individuals, the Enron corporation, Wikipedia, Twitter, and Google News. We uncover bias at di erent levels, even when the data comes from Wikipedia, which has a neutrality and objective writing policy. We conclude with a discussion of the implications of bias that is present in large language models which are being widely used in digital communications for text generation and summarization, automated speech, and machine translation. How can we engage policymakers and developers to enable algorithmic transparency and fairness?