Why Transparency Matters

Why Transparency Matters

In a world in which ‘big tech’ is thriving, there are talk (and concerns) about technology’s role in making decisions, specifically high-stakes decisions. At what point do you know a machine is ready for that responsibility? When do you know it can be trusted with important decisions? Establishing trustworthiness requires the ability to verify and validate, which happens to be the very thing that deep learning models (i.e. black box algorithms) are intrinsically bad at. Imagine someone has an image recognition algorithm that they are using to identify if something is a medication or not; 99% of the time, the algorithm accurately recognizes tablets and capsules as medications, but one day, it labels a tablet as a ‘chair’. What do you do? There is no logical explanation for this. Does this mean that there is a flaw in the algorithm and that maybe it did not learn what you think it learned? If you could not figure out a way to explain why this error occurred, would you still trust the model? In this case, maybe you would still use it, since you can easily catch the mistake; however, what if the model were allowed to make decisions on its own, without human supervision? Would you let it do that?

These are the questions we have to grapple with when we decide if or how we want to use deep learning models to make autonomous, high-stakes decisions in healthcare. This is why the concept of trustworthiness is so important. Among other things, trustworthiness speaks to the idea that we need to be able to verify a model’s performance and reliability before we use the model to make a decision that can impact someone’s health. To be trustworthy, models need to be verifiable; in other words, we need to find a way to explain a machine learning model’s decision. 

As George E.P. Box said,

“All models are wrong, but some are useful.”

Since all models are wrong, it is imperative that we know how they could be wrong, if we hope to minimize risks associated with using them. If we do not know why or how a model came to the conclusion it did, then we do not know its limitations and capabilities. We do not know when it is or is not appropriate to use, or when it will or will not be useful.

Although an imperfect analogy, let’s look at a simple scenario of what can happen when you do not know the why behind something. Let’s say there is a new medication you know nothing about except that you were told that it has to be taken with food (not an uncommon scenario for many patients). Now, imagine a situation where it would be impossible to take the medication with food, because you had to skip a meal or were unable to eat. How do you go about making the decision of whether you should also skip the medication? Is it safe to take? On the other hand, is it safe to skip? Do you have anything on which to base that decision, if you do not know why the medication needs to be taken with food? Think of all the different outcomes that could occur: if a medication needs to be taken with food to minimize non-serious side effects (like GI distress), then taking it without food may result in mild side effects, but the medication will still work just as well. If a medication can cause serious side effects without food, then even though it might be effective on an empty stomach, the risk might outweigh the benefit. If a medication needs to be taken with food to ensure adequate absorption, then taking it on an empty stomach means you will not be able to receive the full effect of the medication, but it also will not cause any harm. Lastly, if a medication needs to be taken with food because food is related to its mechanism of action (e.g. digestive enzymes, drugs targeting postprandial blood glucose), then you likely do not need the medication if you are not eating. Moreover, in some of those scenarios (e.g. rapid acting insulin), taking the medication could result in serious harm. 

Considering all of that, you can think of the above scenario somewhat similar to how you would a black box algorithm. An algorithm gives you an output without any contextual information, such as the instructions “take with food.” That information, however, is brittle. If you do not know the rationale behind that information, how can you integrate the suggestion into a complex clinical decision? How can you apply the information safely across countless unique situations that the model may or may not have accounted for? Remember, models are always wrong. There is no such thing as a model that gets it right every time; and so, when we look at artificial intelligence, and specifically, how machine learning algorithms learn from the data they are given, we need transparency, and that is where the concepts of explainability and interpretability come in. Over the next few posts, we will spend some time diving deeper into these two terms.