Opening AI’s Black Box: How New Explainability Tools Audit Neural Networks

Mechanistic interpretability was named one of MIT Technology Review’s breakthrough technologies for 2026. The field has produced genuinely remarkable results, including the ability to identify individual concepts inside a frontier model and steer behaviour by manipulating them directly.

It has also produced a finding that should temper expectations: when Anthropic applied circuit tracing to Claude 3.5 Haiku, the technique yielded satisfying insight for roughly a quarter of the prompts tested.

That gap — between what interpretability can demonstrate and what it can reliably deliver — is the most important thing to understand about this field, particularly as regulation begins to assume the capability exists.

Why the Black Box Exists

The obstacle is structural rather than a matter of insufficient effort.

Neural networks encode more features than they have neurons. The result is superposition: individual neurons are polysemantic, participating in many unrelated concepts at once. There is no neuron for “Paris” waiting to be found.

This is why simply reading weights tells you nothing. The information is real and it is distributed across a representation that was never organised for human inspection. Interpretability’s central technical problem is disentangling it.

Two Generations of Explainability

The distinction matters, because the two are frequently conflated.

Traditional XAI — saliency maps, LIME, SHAP, attention visualisation — measures relationships between inputs and outputs. Which pixels mattered? Which tokens did attention weight? These methods are established and useful, and they are fundamentally correlational. They also have a troubled validity history: saliency methods notably failed basic sanity checks in a well-known 2018 study, producing similar-looking explanations for trained and randomly-initialised networks.

Mechanistic interpretability attempts something harder: reverse-engineering the internal algorithm, much like decompiling a program. Rather than asking which inputs correlated with an output, it asks which internal components computed it, and verifies that claim causally.

That shift from correlation to causal mechanism is what makes auditing conceivable at all.

The Current Toolkit

Sparse autoencoders (SAEs). The workhorse. An encoder-decoder pair trained to compress activations into a sparse hidden representation and reconstruct them, learning a dictionary of features that are more interpretable than raw neurons. Anthropic scaled these to a production Claude model; Google DeepMind’s Gemma Scope released pretrained SAEs across layers and model sizes, making the technique broadly accessible.

Transcoders. Replace dense MLP layers with sparse, interpretable approximations. MLP layers have been the most stubbornly opaque part of the transformer, so this targets the hardest component directly.

Crosscoders. Map activations across multiple layers and multiple models into a shared feature space. This enables something previously impossible: comparing what two different models represent, which matters enormously for auditing a fine-tuned model against its base.

Circuit analysis. Identifying the specific components implementing a behaviour. The canonical results here — induction heads, the indirect object identification circuit — established that findable, describable algorithms genuinely exist inside these networks.

Attribution graphs and circuit tracing. Tracing how information flows through a model for a given input, producing a graph of what caused what.

Steering vectors and causal interventions. The point where interpretability becomes control. Having identified a feature, you can amplify or suppress it and observe the behavioural change — which is simultaneously the strongest validation method available and a capability with obvious dual-use implications.

What an Audit Actually Involves

In practice, an interpretability-driven audit follows a sequence:

1. Decompose activations into features using SAEs or transcoders. Which features fired for this input?
2. Trace the circuit. How did those features connect to produce the output?
3. Verify causally. This is the step that separates rigour from storytelling — ablate or amplify the component and confirm the output changes as predicted. Correlated components are not explanations.
4. Compare across contexts. Does the same mechanism appear on similar inputs, or was it specific to one prompt?
5. Document and reproduce, so a second team can check the claim.

Step three is where most informal interpretability work falls down. A feature that appears to represent a concept, and a feature that demonstrably causes behaviour associated with that concept, are different claims requiring different evidence.

Where It Still Fails

The honest inventory, and the section most coverage omits.

Coverage is partial. The Claude 3.5 Haiku circuit tracing result — useful insight on roughly a quarter of tested prompts — is a flagship effort by a well-resourced team. DeepMind’s months-long circuit analysis of Chinchilla produced an explanation described as brittle and partial.

Some questions may be intractable. The 2025 “Open Problems in Mechanistic Interpretability” paper, assembling 29 researchers across 18 organisations, explicitly identified that a number of interpretability queries appear computationally intractable rather than merely unsolved.

Not every feature is interpretable. A significant proportion of SAE latents do not correspond to features that are meaningfully interpretable or actionable, which limits how much of a model any given decomposition actually explains.

The validity problem. Research has shown SAEs can produce apparently interpretable features from randomly initialised transformers — networks that have learned nothing. That finding is uncomfortable in the same way the 2018 saliency results were: a method that finds structure in noise cannot, on its own, certify that structure it finds elsewhere is real.

None of this means the field is failing. It means the tools are research instruments, and treating a research instrument as a compliance artefact is a category error.

Chain-of-Thought Is Not an Explanation

Worth separating out, because it is the most widespread misunderstanding in practical AI governance.

When a model produces reasoning before its answer, that text is an output generated by the same process that produced the answer. It is not a log of the computation. Recent research has cautioned explicitly that model-generated explanations, including chain-of-thought, are not reliable proxies for what is actually happening internally, and that mechanistic verification is required instead.

A model can produce plausible reasoning that has no causal relationship to how it arrived at its conclusion. For anyone building audit processes on the assumption that visible reasoning constitutes transparency, this is the finding to internalise.

The Timing Problem

Here is the tension the field is walking into.

The EU AI Act phases in transparency and high-risk obligations between 2026 and 2028. Regulators will increasingly expect deployers of consequential AI systems to explain how decisions were reached.

Meanwhile the tooling explains a fraction of prompts, cannot fully validate its own findings, and is the subject of active proposals to develop guidelines that would make interpretability results auditable in the first place — a field working toward being auditable, being asked to serve as the audit.

Institutional support has also been uneven. Interpretability has attracted prominent advocacy, including a widely-read 2025 essay by Anthropic’s CEO arguing for urgency in the field. It has also seen setbacks, notably the dissolution of OpenAI’s Superalignment team in May 2024, which had included interpretability in its remit.

What to Do If You Deploy Models

Practical positions that do not require solving the research problem:

• Do not claim explainability you cannot demonstrate. “The model provided its reasoning” is not an audit trail.
• Use behavioural evaluation as your primary evidence. Systematic testing across defined scenarios produces defensible documentation today; mechanistic explanation does not.
• Track provenance rigorously — training data, fine-tuning, prompts, model versions. Governance frameworks increasingly expect records rather than explanations.
• Watch crosscoders specifically if you fine-tune. Cross-model comparison is the interpretability capability closest to a practical deployment question: what did fine-tuning actually change?
• Treat steering as a capability, not a fix. The ability to suppress a feature is not the same as knowing the model no longer has the behaviour.

Final Thoughts

Interpretability is doing something genuinely new. Reading concepts out of a frontier model’s activations and steering behaviour by manipulating them would have seemed implausible five years ago.

It is also, on its own published evidence, partial, sometimes brittle, and difficult to validate. The correct posture is neither dismissal nor the assumption that the black box is now open. It is understanding that we can currently see into these systems in specific, hard-won ways — and that the distance between that and an auditable explanation of a production model remains considerable.