Machine Learning Shows us that Systemic Racism is Very Real

With the recent events around George Floyd; IBM, Amazon and Microsoft have all moved out of the facial recognition business in the last week citing the likelihood of them being used for bias Three Big Tech Players Back Out of Facial Recognition Market. They have done this for the same reason I give when asked about bias in ML algorithms, the only way to stop it is to starve the algorithm of the data that would lead to the bias being formed. These algorithms by design can lead to the classification of people by race, sex or any other characteristic that we can tell by looking at people.

Just a quick note on bias vs. racism. The terms systemic racism or institutional racism can probably be replaced with systemic bias. That is to say there may not be a conscious effort to disadvantage a particular group but from the perspective of a member of that group, such a fine distinction is probably immaterial. Either way, they don't get the job or get treated a certain way by the legal system, etc. A claim of, "It's not racism, it's just bias" is likely a cold comfort.

Without going into the technical weeds on how machine learning works, suffice it to say it is modeled after how our brains learn, by forming patterns. Based on a history of data, it makes predictions on future events. For the most part those future predictions are made with the assumption that the status quo will be maintained. That is so say, if A led to B in the past fifty times, A will still lead to B the next time it happens.

Let's take a very simple example of what such an algorithm would look like if we wanted to say, make predictions on what job candidates would be successful in the tech industry. We feed the training algorithm with information from past candidates like, were they hired, what was in their resumes, their names, their pictures, their age, starting salaries if hired, etc. Based on the makeup of today's tech industry, what conclusions will such a model likely come to on what makes up a successful candidate?

- Likely some education and technical experience, that's all good
- White males are more likely to be successful candidates. Uh oh. We can argue how this came to be, but the current industry is dominated by white males and they are currently the most successful candidates, so that will be picked up
- In some cases minorities and women can be paid less doh!
- People with "white and male sounding" names are more likely to be successful
- People who live in more affluent neighborhoods and suburbs make candidates more likely to be successful
- Most viable candidates are likely between, eh 30-50 years old

Feed the trained model data on new candidates and all other things being equal, the model is likely to conclude that 30-50 year old white males are the ones to hire. Now is that model racist? Is it biased? Are the people using the model even aware of how it is coming to its conclusions under the hood? I'm not sure the answer to any of these questions is per se important. What is important are the results.

Edit to add: based on some feedback, I want to make sure to point out my hiring in the tech industry example is not completely theoretical. Amazon scraps secret AI recruiting tool that showed bias against women

How does this relate to systemic racism? It relates because these models learn in very similar ways to how we learn. Our learning and experience, consciously or unconsciously, trains us to reinforce the status quo. Past experience is what our minds use to give us conclusions for future events. If our past experience shows us that it is mostly white males are who we interacted with successfully in the tech industry, we will form the impression that is what will most likely work in the future. Not necessarily purposely, or even consciously, but that's the most likely conclusion that our brains will reach, regardless of your race or sex.

Our minds can't do what I suggested to students with bias, that is to say we can't simply starve our brain of the data that would lead to biased outcomes. This is why people that say they "don't see color" should rethink their position. Like my hypothetical job applicant model above, we definitely see color, race, sex, you name it. Unlike ML that can be starved of certain data, we can't choose to not see it or not hear it. All kinds of factors are being fed into our brains every single day and in every single interaction.

The important thing to realize is that without conscious effort to change things, our minds will default to coming to conclusions that are supported by past experience. What is successful today is what will be successful tomorrow. Problems I see today will be the problems I see tomorrow. Like machine learning, it's just the way our minds work. If we are not conscious of the fact that our minds are coming to conclusions based on that data then we will forever be locked into our current cycle. We have to be very cognizant of what is happening and why we are likely getting certain impressions on people. It needs to be a conscious effort to really analyze what data we have available to us on a situation by situation basis. If we just allow our minds to stick with their first impressions, there may be all kinds of immaterial biases at play.

BTW, all of our minds work like this. This means that minorities and women are just as likely to be unconsciously biased against themselves. The more I work with machine learning, the more inescapable this conclusion has become.

Machine Learning and the Expectation of Mistakes

Traditionally in software development we've considered anytime an application creates an incorrect result to a bug. A bug is considered an error in a system; not that the application simply made a mistake, but that there is a fundamental flaw in the system itself. Why is this important? Because mistakes are something that can be learned from and can be self corrected while bugs are fundamental flaws in the system where the only way they can be corrected is by outside intervention to change the operation of the system itself.

Last year Elaine Herzberg was struck and killed by an Uber self driving car. Was this a bug in Uber's software or was it a mistake that the software made? This is an important distinction and one for us to recognize is a real question. Traditionally most applications do exactly what the code tells them to do, no less, no more. But machine learning changes that. We have to ask ourselves if the software in that car took the wrong action based on the data that had been used to create the model to date; if the results of this action were fed back in would the updated model in same situation make decisions that lead to a positive outcome (no crash). If the answer to that is no, the base software needs to change, then we have a bug. But if the answer is yes, we have a self driving car that made a mistake based on a new situation that past experience didn't prepare it for and it may not happen given the same situation a second time.

To do complex automation tasks like autonomous vehicles, where the application will encounter unexpected situations, requires machine learning and the concept of mistakes to be successful. To treat all unexpected outcomes as bugs is not sustainable, there is no way that developers will be able to code in every situation into an application a self driving car may encounter or keep updating it as new situations are discovered. But if we are willing to treat them as mistakes that can be learned from with the model updated based on new situations then we have something that is within the realm of reality.

Software that doesn't just have bugs but also can make mistakes leads to some interesting mental shifts that we need to make as an industry and in society at large. Any application that uses machine learning internally should take a look at any unexpected outcomes and ask themselves the fundamental question, "Is there a mechanism to feed back results into the system whereas the model can be updated and lead to a better (or at least different) result next time?" If the answer is yes then no other action may be required other than monitoring or recreating the situation to see if better results are created. This is a big shift in our thinking.

It is also a big shift for society. Families of people killed by self driving cars may not want to hear that the software made a mistake. They likely want 100% accuracy in correct decision making. A standard that we don't apply to people and self driving cars won't be able to meet it either. There are some other and obvious legal and liability questions that will come to play as well.

I can also see that organizations that don't do well with their people making mistakes, likely are going to have a hard time accepting software that can make mistakes as well. They will categorize them as bugs, just like they do when their people make mistakes. This will be another area where organizations who are more accepting of mistakes will likely have a competitive advantage as their culture will be more accepting of the fundamental learning process that machine learning requires.  Requiring the model to be perfect out of the gate is just not a realistic situation in a complex world with nearly infinite possibilities and situations.

So what should organizations keep in mind when starting with machine learning:

- Understand there is a difference between a bug and a mistake.
- Software that has no way to feed back in the accuracy of the model results can only have bugs. Mistakes require the capacity to learn, to improve.
- Mistakes require different remediation action than bugs. Checking to see if a model learned from the incorrect outcome may become a QA question. If it is found that the model can't learn from the situation, then that would become a bug; a flaw in the ability of the system to learn.
- Machine learning models deal in probabilities. The model may return that it is 95% sure a picture has my face in it. Is 95% good enough? Probably, the threshold of acceptability will need to be defined somewhere. But that should not be the end of the discussion. The best applications will have a way to feed back into the model if the results are correct or not so next time the same picture is guessed more accurately.
- Mistakes need to be tolerated, not just by the development team but the management team, the users of the applications and anyone in society at large that interacts with them. Mistakes may feel like the personification of software systems, but the mental shift is recognizing that a system that can learn from bad outcomes has the capacity to make mistakes and requires no self awareness.
- Many models, particularly ones created with deep learning, are black boxes. They cannot be fixed the same way application code can.
- Without allowing for the concept of mistakes, our current software development capabilities will not be able to create systems that can effectively work in complex environments with uncounted and unknown situations they may experience.

The rise in use of machine learning has the capacity to automate things that we have never been able to automate before. One of the likely victims of that upheaval is the understanding that all problems are flaws or bugs. We will have to be more accepting of mistakes, just as we are of our fellow humans.