Leveraging visual representations to understand AI behavior

By leveraging visual representations, a black-box-like AI behaviour can be demystified, such as how AI makes automated decisions, whether AI uses unnecessary or insignificant data, or which biases in data are persistent.

* Source: https://www.geeksforgeeks.org/what-is-saliency-map/

A heatmap (or saliency map) is a popular visual representation for generating user attention data during traditional usability testing with eye trackers.

Rather than showing what users are actually paying attention to on your AI products, AI can employ heatmap-like visual representations to exhibit its decision patterns.  As illustrated in the image below, the regions marked with red are the most critical area for recognising an object (here, a lion, dog, or bird).

The heatmap indicates whether the respective data are sufficiently helpful for learning and whether unnecessary elements are used to make predictions. Hence, visual representations can also determine the points requiring human involvement to make better decisions and the measures used to secure the reliability of AI behaviour.