// AI governance · Agentic systems
Explainability tells you what a model tends to do. Replayability tells you what your institution actually did — this decision, this context, this moment — and whether it matched what was authorised.
Most AI governance frameworks were designed for models that recommend. Institutions are now deploying systems that decide, at a speed no review cycle can follow. When one of those decisions fails, the question a board faces is not why the model behaved that way. It is simpler and harder: can we reconstruct what happened, and can we prove it was authorised?
If you cannot replay a decision, you should not automate it.
Explainability tools — LIME, SHAP, and a generation of interpretable models — are genuinely useful at the boundary of a single model. They answer one question well: why did this model produce this output?
That question stops being the important one the moment a decision crosses systems. An agentic decision passes through a policy, a rule engine, a model, an orchestration layer and an execution system, often owned by different teams. Explainability can account for one component. It cannot tell you whether the decision was authorised, whether the rule applied was the rule in force at that moment, or what changed as the decision moved between agents.
The EU AI Act, the NIST AI Risk Management Framework and financial-sector guidance from authorities such as the Monetary Authority of Singapore establish real expectations for risk management, accountability, human oversight and record-keeping. Those expectations are necessary. They do not automatically produce a replayable institutional decision. A bank can retain full model documentation and complete system logs and still be unable to reconstruct how a policy, a rule version, a human exception and an AI agent’s action combined to produce one customer outcome. Replayability turns governance evidence into a decision-level capability.
In both Silicon Valley Bank and Knight Capital, the output was observable. Nobody needed a model explained. What could not be reconstructed was the chain that produced the output — and that is precisely what explainability does not cover.
// The distinction
For a single decision followed step by step through all four standards, see the worked example: What does it mean to replay a decision?
| Explainability | Traceability | Replayability | |
|---|---|---|---|
| What it answers | Why did the model produce this? | What path did this decision take? | Was this decision authorised — at that exact moment? |
| Unit of analysis | Single model output | Decision path, one system | Full causal chain across all agents |
| Temporal scope | Post-hoc only | Post-hoc | Real time and post-hoc |
| Cross-system coverage | No | Partial | Yes |
| Intent binding | No | No | Yes |
Intent binding is the row that matters. It is the difference between knowing what a system did and knowing whether it was permitted to.
In documented institutional failures, the original error is rarely the largest cost. The larger share accrues during the period in which the institution cannot reconstruct, analyse or stop what follows.
Knight Capital, August 2012. In forty-five minutes: more than four million orders sent against 212 customer orders, 397 million shares traded, and losses exceeding $460 million — approximately 75% of the institution's equity capital. The output was visible throughout. The link between the authorised intent and the executing code was not.
Silicon Valley Bank, March 2023. A disclosed $1.8 billion loss on held-to-maturity securities was followed within 48 hours by $42 billion in withdrawal requests. The decisive variable was the speed of amplification measured against the institution's ability to see what was happening.
Cases drawn from public regulatory filings, official investigations and published research. Analysis is educational and is not investment, legal or regulatory advice.
// What replay requires
Replaying a decision means reconstructing every layer it passed through — not a log of what happened, but a record of what was authorised, what rule was in force, where judgment entered, and who or what held the authority to act.
Those layers have names. The Decision Integrity Chain™ (DIC™) maps the eight layers a decision passes through inside an institution:
Purpose → Strategy → Intent → Rules → Judgment → Decision → Outcome → Feedback
Where a layer cannot be reconstructed, the chain is broken at a specific, identifiable join. That is the practical value of naming them: a failure stops being "the AI got it wrong" and becomes "Layer 4 was never encoded, so Layer 5 had nothing to apply."
This is the work Decision Engineering™ does — examining whether institutional purpose, intent, authority and accountability remain intact as decisions travel through human and automated execution.
Choose one automated decision your institution made last quarter. Can you produce the authority under which it was taken, the rule in force at that moment, the data that shaped it, and the person who remains accountable for it?
If that takes longer than an afternoon to assemble, the institution is operating on the assumption of control rather than the evidence of it. That assumption is only tested when something has already gone wrong.
// Common questions
Replayability means being able to reconstruct not only what the system did, but what it was authorised to do — and to detect the difference while action is still possible. It asks what the rules in force at that moment were, the authority under which the action was taken, and where execution departed from what was authorised. It makes no claim about determinism and does not require an identical model output to be reproduced. Explainability describes how a model tends to behave. Replayability establishes what the institution actually did on a specific decision, at a specific time, and whether that matched what was authorised. For agentic systems acting at machine speed, it is the only standard that supports accountability after the fact.
Explainability tools such as LIME and SHAP are genuinely useful at the boundary of a single model. They answer why one model produced one output. They cannot answer whether the decision was authorised, whether the rule applied was the rule in force, or what happened as a decision passed between several agents and systems. In both Silicon Valley Bank and Knight Capital the output was observable. The failure was in the governance chain that produced it, not in the output itself.
Three things most frameworks currently omit. Intent binding, so that every automated decision carries the authority under which it was taken. Cross-system coverage, because agentic decisions cross model, platform and team boundaries. And real-time reconstruction rather than post-hoc review, because at machine speed a decision can be repeated thousands of times before a quarterly review convenes. Explainability offers none of these. Replayability requires all three.
To replay a decision you need every layer it passed through: Purpose, Strategy, Intent, Rules, Judgment, Decision, Outcome and Feedback. The Decision Integrity Chain™ names those eight layers, which is what makes replay possible in practice rather than in principle. Where a layer cannot be reconstructed, the chain is broken at a specific, identifiable join — and that is the point at which control becomes an assumption.
In documented failures the original error is rarely the largest cost. The larger share comes from the period in which the institution could not reconstruct, analyse or stop what followed. Knight Capital lost more than $460 million in forty-five minutes — about 75% of its equity capital. Silicon Valley Bank faced $42 billion in withdrawal requests within 48 hours. In both, the decisive factor was the speed of amplification against the institution's inability to see what was happening.
Every term used across this work — replayability, the Decision Integrity Chain™, the Fiduciary Gap™, decision drift — is defined once in the glossary.