The Power of Chaos

How the race to squeeze more juice out of AI is quietly building a wall of unprovable decisions - and why the missing accountability ratchet is what will decide who wins in court.

The juice is real

I want to be clear about this before anything else. AI is producing outcomes that were not achievable five years ago. A radiologist reading scans with a model catches cancers eighteen months earlier than she used to. A biotech shop searches chemical space in a week that would have taken a decade of graduate students. A hiring platform sees signals in resumes that a human recruiter under time pressure would never register. A trading desk finds statistical edges in tick data that no analyst could hold in her head.

This is the power. The juice is real.

But every one of those wins carries a second-order effect that almost nobody is building for. The systems generating those wins are not keeping the records that would let anyone — regulator, jury, opposing counsel, the person who was denied a job or a loan or a proton-beam treatment — reconstruct what happened at the moment the decision was made.

That is the chaos. And the chaos is starting to arrive in courtrooms.

---

Five examples of how the juice and the chaos come from the same source

1. Pharma discovery

A generative chemistry pipeline proposes ten thousand candidate molecules for a target. The team narrows to fifty, then to five, then to one that goes into IND-enabling studies. Three years later the drug is approved. Five years later a plaintiff sues alleging the compound's off-target effect was foreseeable and that the original candidate set should have been filtered against a known safety signal.

The company can produce the final molecule. It can produce the IND package. It can produce the meeting minutes from the discovery team.

What it cannot produce is a record showing, at each generation step of the model, which safety filters were in force, which training corpus was loaded, and which candidate was scored against which structural-alert catalog. The model was retrained six times during the discovery cycle. Nobody logged which version was in production when the winning candidate was scored.

The company knows in its bones that the safety filters were on. It just cannot show a jury, decision-by-decision, that they were on for the specific molecule that hurt the specific plaintiff.

2. Diagnostic radiology

A hospital deploys an AI-assisted mammography system. It catches early-stage tumors that would have been missed. The hospital's outcome data is better than any peer institution's. Patients are alive who would not otherwise be alive.

Two years in, a patient sues alleging that her missed diagnosis was the result of the AI system deprioritizing her scan below the human reader's queue. The hospital defends: the model was operating within its validated envelope, the human reader was the ultimate decision-maker, the system was FDA-cleared.

Discovery arrives. Plaintiff's counsel wants the record showing, for the specific scan on the specific day, which model version was in production, which confidence threshold triggered the deprioritization, which quality-control policy governed the reader's queue, and what the reader saw on her monitor at the moment of triage.

The hospital can produce the model. It can produce the scan. It can produce the reader's report. It cannot produce the Layer-2 tie that binds those artifacts to the governing standards in force at the moment of triage. The vendor pushed a model update on the 14th. The QA policy was revised on the 22nd. The scan in question was on the 18th. Which combination was actually in effect? Nobody logged it.

The plaintiff does not have to prove the wrong combination was in effect. The hospital has to prove the right one was. Without the ratchet, that proof does not exist.

3. Algorithmic trading

A quant desk builds a signal that produces excess returns. The desk deploys it. It works. Regulators eventually notice a pattern of trades that look, to an examiner, like the desk may have been trading ahead of information that was not yet public.

The desk explains: the signal was generated by a model trained on public data only. Show us the training data. The desk pulls it up.

The examiner asks the harder question: which version of the model was in production at 10:32:14 on the trade in question, what feature-set was the model consuming at that moment, and what data-source feed timestamp was the model processing?

The desk retrained twice during that quarter. Feature-store versions were rolled forward twice. The data-feed vendor did a schema change in the middle. No one logged, per-trade, which combination was in force. The evidence trail exists in fragments across three engineering teams and two vendors. Reconstruction takes six months, produces a probabilistic answer, and satisfies nobody.

The chaos: the desk did nothing wrong. It just cannot prove that it did nothing wrong.

4. Hiring and screening

A company deploys a screening tool that improves quality-of-hire and reduces time-to-fill. Every metric moves in the right direction. Two years later, the EEOC opens an inquiry about a pattern of disparate impact on candidates over forty.

The company produces the vendor's fairness audit. It produces its own aggregate hire-rate data. It produces the model card.

The EEOC asks: for the 3,400 applicants who were screened out in the third quarter of 2024, produce a per-decision record showing which model version, which feature set, which threshold, and which bias-mitigation policy was in force when each decision was rendered.

The vendor changed the model twice that quarter. The company adjusted the threshold once. The bias-mitigation policy was updated. No per-decision Layer-2 record exists. The company is left arguing from aggregates that a plaintiff will characterize as after-the-fact reconstruction.

5. Legal research

An attorney uses an AI research tool to draft a brief. The brief cites a case. The case exists. The pinpoint citation is right. The proposition of law is right.

Three years later the client sues for malpractice, alleging that a controlling authority was missed and that the attorney should have caught it. The attorney's defense depends on showing what her research tool actually surfaced on the day she drafted the brief, and what the state of the law was in the tool's index on that day.

The tool has been updated dozens of times. The index has been refreshed continuously. There is no snapshot of what the tool would have returned on the day in question. The attorney cannot reconstruct the search she actually ran. The plaintiff argues that a diligent attorney would have found the missed case; the attorney argues she used industry-standard tools; nobody can prove what those tools actually returned.

The attorney did competent work. She just cannot prove the work she did.

---

The pattern

In every one of these examples, the chaos is not the result of anyone doing anything wrong. The chaos is the result of the field's obsession with squeezing the next percentage point of juice out of the model — retraining faster, updating features more often, rolling in new data sources, pushing new versions to production — without pausing to build the accountability skeleton that other regulated industries have carried for fifty years.

Pharmacies keep a record, per compounded batch, of which standard operating procedure was in force. Airlines keep a record, per maintenance action, of which airworthiness directive was applicable. Nuclear plants keep a record, per operation, of which technical specification governed. Financial auditors keep a record, per signed statement, of which auditing standard applied.

None of those industries stopped innovating. They innovated with the accountability skeleton in place. The skeleton is not the enemy of progress; it is the condition of trust that lets progress continue.

AI is not doing this. AI, in its current adolescent form, records what it produced (Layer 1) and sometimes signs a general assurance that things were done properly (Layer 3). It does not record, per decision, which specific policies, model versions, feature sets, thresholds, data sources, and mitigation rules were in force at the moment the decision was rendered. That middle layer — Layer 2 — is where the accountability tie sits in every mature regulated industry, and it is exactly the layer AI systems are missing.

---

The ratchet as arbiter of fairness

When the ratchet is missing, the party without the records has to prove a negative. That party will lose, or will pay to make the case go away, regardless of the merits. That is not fair. It is also not what a mature legal system is supposed to produce.

When the ratchet is present, both parties can argue about what the decision was, what the policy said, whether the policy was appropriate, and whether it was followed. That is a legitimate argument on the merits. It is what courts are built for.

The ratchet is not a burden on AI innovation. It is the mechanism that lets AI innovation survive the litigation that is already arriving. It is what allows the pharmaceutical company to defend its discovery pipeline, the hospital to defend its triage system, the trading desk to defend its signal, the employer to defend its screening, the attorney to defend her brief.

It is the arbiter of fairness. Without it, courts will be forced to allocate loss to whichever side has fewer records — which, right now and for the foreseeable future, is the AI-deploying defendant. With it, courts can do what they are supposed to do: hear the arguments on both sides and decide on the merits.

---

The choice

Every organization deploying AI right now is standing at the same fork. On one path, keep squeezing the juice, keep shipping model updates, keep retraining, and hope no one ever asks the Layer-2 question. On the other, build the ratchet now, at the cost of some engineering discipline and some storage, and be prepared to defend every decision that ever comes back around.

The organizations that pick the first path will produce more juice this year and next. The organizations that pick the second path will still be operating in the courtrooms of 2029 and 2032. That is the trade.

The power of AI is real. So is the chaos. The existence of the 3-layer ratchet is what decides which one wins.