When probability meets accountability

Digital Agile Enterprise Architect & Next Generation Architecture Chapter Leader FMRC
Consider two people applying for the same service. Their relevant circumstances are identical, the same policy applies, and they submit the same information to the same system.
The first application is approved. The second is rejected. Our AI system can explain both decisions.
I would still say we have a problem.
This thought experiment gets to something fundamental about using generative AI for consequential decisions. We spend a great deal of time asking whether AI can explain an answer. But we also need to ask whether the answer should have been the same in the first place.
Probability is useful
Large language models generate responses probabilistically, giving them capabilities conventional software lacks. If I ask AI to interpret an ambiguous message, examine a complicated document or work through an unfamiliar situation, you may want that flexibility. The problem doesn't have a single path that can be fully specified beforehand.
Regulated business processes contain plenty of this kind of uncertainty. Applications arrive with free text. Customers phrase things differently. Relevant information hides in supporting documents and cases don't always fit a convenient template. AI can help make sense of that material.
Eventually, though, the process may reach a different kind of question. Perhaps legislation defines eligibility. An insurance policy specifies whether a particular condition is covered. A compliance rule determines what happens when established criteria are met.
The system has moved from interpreting the case to applying a decision that the organization has already defined. How much probability do we want there? I assume none.
An explanation may not be enough
Explainability is important, especialy when automated systems influence consequential decisions. A customer may want to understand a result. An employee may need to review it, while an auditor or regulator may need evidence of how the decision was reached and which laws and regulations it was based on.
Our example of the two applicants exposes a weakness in treating explanation as the entire problem. If a system can produce two different answers from the same relevant facts and then explain both answers convincingly, providing the explanation has not resolved the inconsistency.
For certain decisions, you require something more specific:
- Which exact information did the system use?
- Which rules, based on certain laws and regulations, are applied to the decision?
- Which version of the decision and of its rules were executed at the time?
- How did these inputs produce this particular output?
And if I submit the identical validated facts to the same decision again, can you reproduce the exact identical decision outcome?
That question has direct consequences for the architecture.
Find where uncertainty ends
Imagine an application consisting of a form, correspondence and several supporting documents. An AI system could examine the material and identify case-relevant information. It could classify documents, extract candidate facts and flag something that appears to be missing.
Those are useful roles precisely because the inputs are messy. Once the system establishes and validates the relevant facts, a deterministic service can apply the approved eligibility rules. The system can record the facts provided to that deterministic service, the rule versions and the result returned.
This separation does not remove risk. If AI extracts the wrong facts from a document, software can faithfully calculate the wrong answer from those facts. We still need appropriate validation, escalation, and human oversight.
But it does give us a clearer view of where uncertainty exists and where consequential decisions are made. When something goes wrong, we have a better chance of understanding what happened.
Regulation sharpens the question
These issues become especially visible in regulated environments, where organizations may need to show auditors how systems are controlled and how consequential decisions are made. The exact requirements vary by sector, jurisdiction, and use case, so architecture teams need the relevant legal, compliance and risk expertise when evaluating or designing a real application.
I would ask the reproducibility question even when regulation didn't require it. A customer can challenge a decision months after it was made. A rule may have changed in the meantime, and the organization may need to reconstruct what happened using the information and logic that applied on that particular day. "I asked the model again and it gave me this" is unlikely to be a satisfying audit trail.
A simple test
When examining a decision inside an AI-enabled process, I like one question because it forces us to think about the nature of the work:
If nothing relevant has changed, are we comfortable with the answer varying?
There are situations where the answer may genuinely be yes. A different proposed response to a service problem may be equally good. A summary can be phrased differently. More than one plan may achieve the same objective.
On the other hand, if a different answer would create a serious problem, especially when human lives are involved, we have learned something important. The decision requires explicit, deterministic logic, stronger validation, a controlled service for the agent to call and a human in the loop.
AI offers powerful ways to work with ambiguity, even in regulated operations. Still, some decisions require repeatability with identical input and 100% conformity to laws and regulations. The architecture must preserve that capability without letting ambiguity leak into decisions expected to consistently produce the same results with the same input.
Our two applicants deserved an identical decision because nothing relevant differed between them. The architecture should know that too.
We’re looking forward to our sponsorship at Adopt AI 2026, December 3-4, in Paris. Each Thursday leading up to the event, Agile & Digital Enterprise Architect Peter Kalmijn, a member of the Future Makers Research Community (FMRC) at Atos, will explain why good AI architecture starts by saying no.

