Explainable AI – Start with the Data
When a machine learning (ML) model is used for decision making, we want to know why a certain decision has been made. We envision an AI system with an ML model at its core for which it can generate an explanation for each output. This leaves use with two obvious questions:
- What is an explanation?
- Is it possible to create an explanation for the output produced by an ML model?
I have discussed the first question in an earlier blog post. In a nutshell – what counts as an explanation to me might not count as an explanation to you. Whether an explanation is acceptable depends on context and recipient. In the guidance jointly produced by the ICO and the Alan Turing Institute the term rationale explanation is used to refer to the reasons leading to a decision which should be delivered in “an accessible and non-technical way”.
Looking at the second question, can an AI system craft some language explaining the output in a way that takes into your account your domain understanding and the decision context? Alas, it cannot. When the ML model was created you were not in the picture. It doesn’t know anything about you or the current context apart from the input data that has been given to it to compute a decision for you. Current AI research is nowhere near to produce what could count as a rationale explanation. All currently available methods to explain the output of an ML model are limited to very technical attempts that are only useful to experts.
If we cannot rely on an AI system to produce meaningful explanations, what can we do? An ML model is a condensed representation of the training data used to create it. The representation is based on similarity. Similar data is represented in the ML model is a similar way leading to similar outputs. An ML model making a decision about you essentially says: your input data is most similar to some other data I have seen before and have used to create an internal representation which in turn I have linked to a decision that has been mostly right.
A good set of questions to ask the ML model would be: what is that other data you have seen and why are you saying my data is similar to it? Where does this other data come from and have you checked that it is correct and of good quality? Have you checked it for bias?
Since the decision is based on data similarity we can further ask: is the similarity based on reasonable features meaningful to the problem domain? Is my data similar to a sufficient number of other cases or is it similar only to an outlier? How often did the given decision apply to the cases my data is similar to and how often was a different decision correct?
All these questions can be answered by the developer of the ML model and all answers begin with the data used to build the model. Answering these questions requires the ML engineer to study and document the data in relation to the application domain and follow a best practice engineering process embedded in governance that provides quality assurance.
Using current explanation techniques as part of the engineering process can help the ML engineer uncover serious flaws in the trained model like, for example, bias or cheating.
In the future we may be able to build an AI system that has sufficient introspection, context understanding and empathy to craft meaningful language for explaining decisions. I won’t hold my breath though. In the meantime, let’s shift focus from the ML model to the data that has been used to build it. Let’s shift focus from technical methods that generate some form of post-hoc technical understanding to explaining the raw material that is right in front of us before we begin building a model.
Detlef Nauck, Head of AI & Data Science Research, BT
