Displaying the feature maps and convolutional filters are a great way to explain the performance of our CNN layers. But what of our dense multi-layer perceptron architectures? How can we explain to stakeholders 'how the magic happens' inside these structures?
Enter 't-SNE'; short for 't-distributed Stochastic Neightbour Embedding'. This technique takes very high dimensional data and reduces it to 2D or 3D. It does this by preserving Euclidean distance. Points that are 'close' in high dimensions are thus close in low dimensions.
This allows us to plot the activations in hidden layers as points in a 2D or 3D chart. The hidden layer;s might have hundreds, thousands or tens of thousands of neurons. When seeing certain types of image (e.g. cats) certain patterns of neurons will be active, while others will be inactive. When presented with different images (let us say, ships) then there will be different patterns of active and inactive neurons.
By preserving the euclidean distance between vectors in high dimensions and their corresponding points in 2D or 3D we can see clusters of activations in response to specific images.
While this all sounds dry and complicated (frankly it is) we explore a mechanical intuition for t-SNE by imagining all our points connected by springs and magnets. Springs can repel (if compressed) or attract (if stretched). Magnets can repel (same polarity) or attract (different polarity). The force on the spring will be in proportion to distance; magnetic force is proportional to the inverse square of the distance.
We see how these virtual springs and magnets attached to each point balance out to give us our characteristic t-SNE plots. The clusters that are formed by these spring and magnet forces group objects that are similar. They allow us to show characteristic patterns of neurons firing in response to the stimuli of specific objects. Again - when show a cat a specific pattern of activations are observed, which are very different from the activations when a neural network is stimulated with and image of a ship.