How Prejudice Creeps into AI Systems
One challenge for systems powered with Artificial Intelligence (AI) is the biases that may be embedded in the algorithms. In my previous article, I explained the inner processes which take place when AI goes rogue. In the following, I will deepen the question where these biases actually come from – and how those sources are different from well-studied bias problems in conventional technologies.
Machine Learning (ML) algorithms identify patterns in data. Their major strength is the desired capability to find and discriminate classes in training data, and to use those insights to make predictions for new, unseen data.
In the era of "big data", a large quantity of data is available, with all sorts of variables. The general assumption is that the more data is used, the more precise the algorithm and its predictions become. When using a large amount of data, it clearly contains many correlations. However, not all correlations imply causation. No matter how large the dataset is, it still only remains a snapshot of reality.
Let's take an example:
In a training data set on claims of a car insurance, red cars may have caused more accidents than cars of another colour. The ML algorithm detects this correlation. However, there is no scientific proof of causation between the colour of a car and the risk of accidents.
Solely for the sake of the algorithm's performance it is crucial to notice and eliminate this kind of unwanted correlations. Otherwise, the algorithm is biased and results on new data in production may be poor. In the case of the example, a competitor with a better algorithm, which does not falsely attribute a higher risk to drivers of red cars, can offer a lower price to those customers and entice them away.
Beside the performance aspect, there is a second, even more severe problem which appears when the predictions impact people, and when the algorithm is biased to favour privileged groups over unprivileged groups, resulting in discrimination.
It is important to note that these unwanted discriminations may happen without explicitly providing sensitive personal data. In fact, other attributes can implicitly reveal this information serving as proxy. For example, a car model can hint at the owner's gender, or the zip code may correlate with a resident's ethnicity or religion.
Where does it come from?
The problem of bias is obviously not new. Even before the emergence of AI, plenty of different forms of bias were known to possibly cause unwanted and unexpected results in technical systems. Automation bias for example is what happens when people trust suggestions of automated systems over human reasoning. Several severe airplane accidents happened in the past because the pilots had trusted the autopilot over their own judgment. Another type of bias may occur when an algorithm is deployed in an environment for which it was not trained in the first place. For example, if it is applied in a different geographical region or on a different group of people.
While explicitly programmed rules in algorithms or the way they are used in practice may produce biased results, this is a long-known problem which already applies to conventional deterministic algorithms. In this article, I focus on the new sources of bias, which gain in importance with the rise of machine learning technologies. More specifically, I discuss human bias in data and selection bias.
Human bias
The first source of bias which comes naturally to mind is human bias. Different types of this kind of well-studied bias are outlined below.
Training data can consist of labels of objective observations, as for instance coming from a measuring device. However, training data may also involve human assessment. Data labels which include human judgment may have been labelled with prejudice. Since the labels serve as ground truth, the algorithm's performance directly depends on them, and any bias contained gets reproduced at scale in the model. So basically, a well-studied form of bias finds its way into a new technology here.
Selection bias
Another, less obvious source of bias is the process of how the data are collected. If the data do not reflect the real distribution, a ML algorithm which is using it for training will learn and enforce the bias. The table below provides a list of different types of biases which may cause selection bias in data.
So what
ML algorithms are strongly dependent on the data they use to create the predictive model. These training data may be biased, in particular in form of human biases or selection biases. Because the algorithms are prone to any such effects, and due to the potential of getting deployed at scale, even minimal systematic errors in the algorithms can lead to reinforced discrimination.
Many thanks to Antoine Pietri for his valuable support in writing this post. In my next article, I explain why deleting the sensitive attributes is not a simple solution to AI fairness.
