← All finance signals
Governance

The Explainability Demand

People stop accepting the black box, refusing any verdict a machine cannot look them in the eye and explain.

What is it?

People increasingly refuse an automated verdict — a hiring rejection, a loan denial, a diagnosis — carrying no legible reason. Explainability is becoming a mandatory design constraint, forced by regulators and users who want more than an outcome. Whether a plausible reason is the same as a true one stays open.

Why now?

Regulation is catching up to systems already in deployment — the EU’s AI Act now requires documentation and disclosure for high-risk models, and law is doing what engineering hasn’t: forcing legibility as a condition of use. The harder problem sits underneath: the most capable models are often the least interpretable.

Who’s building it?

Deployers of hiring, lending and diagnostic systems are building the explanation layer — but regulators are setting its bar.

Deployers of AI decisionsInterpretability vendorsEU regulatorsCompliance auditorsAffected users
The full read
Why now, who is building it, where the market goes, and what stays unsettled.
Get weekly →
01 · The market
Who’s in it
Builders — deployers of decision models, and interpretability toolmakers beneath them.
Buyers — employers, lenders and hospitals adopting these systems, needing defensible answers for regulators.
Gatekeepers — regulators, courts and auditors deciding what counts as a real explanation.
What’s at stake
Compliance and capability pull against each other — the most capable systems are often the hardest to explain.
Legal exposure grows wherever a decision cannot be explained.
Documentation now doubles as evidence in court and audit.
What’s unsettled
Whether post-hoc explanations are actually faithful, or just plausible-sounding cover for regulators.
Who certifies an explanation as good enough to trust?
Whether frontier labs absorb interpretability in-house, or a separate audit industry grows around them.
02 · How the demand matureda progression, not a timeline
Disclosure
Regulators mandate documentation first — a paper trail, not yet real legibility.
Interpretability tooling
Vendors build explanation layers to satisfy the paperwork, whatever their accuracy.
Contestation
Users start challenging explanations in court and complaint, testing whether the reasons given actually hold.
Standardisation
Explanation formats settle into something buyers can compare, and regulators can actually audit.
03 · Where the market goesthe forks, not a prediction
→Converge or fragment — one shared explanation standard across sectors, or hiring, lending and health each end up with incompatible rules of their own.
→Frontier labs may quietly standardise explanation methods before any regulator manages to specify one.
→A dedicated interpretability-and-audit industry could form, or the frontier labs absorb the whole function in-house.
→Explanation could stay legible-but-shallow, satisfying audits without ever answering the harder question.
The demand assumes a plausible-sounding reason and a true one are the same thing — and for the most capable models, that has never been shown.
04 · The Compliance-as-Feature deskadjacent markets, moving now
Regulated Reality
Deepfake law spreads state by state.
30 US states now have election-deepfake laws (2026).
Public Citizen
The Bias Audit
Auditing algorithms for discrimination becomes a regulatory requirement.
Colorado repealed its AI Act, replaced it (2026).
Colorado General Assembly
The Liability Question
Courts start weighing responsibility for un-explained AI harm.
US court let Character.AI product-liability claims proceed (2025).
National Constitution Center
05 · The Wire
01
EU deal delays high-risk AI duties but keeps transparency date
Gibson Dunn ↗
02
South Korea guarantees explanation for automated decisions
Chambers and Partners ↗
The panel on this signal
They agreed on
–Legibility is becoming mandatory, but most explanation methods are post-hoc rationalisations, not proof the model actually reasoned that way.
–An audit-and-documentation market is forming around the requirement.
–Regulators, not engineers, are setting the bar for what counts as an acceptable reason.
The ML systems engineer disagreed on
–Post-hoc explanation methods optimise for human-legible narratives about a model’s decision, which is a distinct technical target from faithfully representing the actual computation; regulatory mandates for explainability do not guarantee that gap between legibility and fidelity closes.
Know more about the method →
✦How this signal was built
01 · Panel
A Delphi panel of 4 AI analysts, each reasoning independently.
→
02 · Debate
They challenge and revise each other’s reads over two rounds.
→
03 · Evidence
Every claim is tied to a cited, linked source on The Wire.
→
04 · Synthesis
A moderator writes it into one read, disagreement and all.