What a decade of logging incidents says about the questions nobody can answer.
In 2018 Twitter disclosed that a bug had written user passwords into an internal log in plaintext, before the hashing step completed. GitHub disclosed a near-identical failure the same year, caused by an anti-spam system rather than the authentication path.
Both companies found the problem themselves. Both disclosed it. Neither is a careless engineering organization.
That is the part worth sitting with. If two of the most scrutinized engineering teams on earth were storing something they had designed themselves never to store, and did not know until they went and looked, then "our system does not retain that" is not a statement anyone can make confidently from an architecture diagram.
It is a claim about what the fielded system actually did, on a particular day, under a particular configuration. And it is not a capability most organizations have ever been asked to build.
Somebody asks: does your system keep X?
The natural answer comes from design. The data flow goes here, it is transformed there, it is written to this store with a ninety-day retention, and X is not among the fields persisted. Somebody draws the diagram. Everyone agrees.
The diagram is not wrong. It is a description of intent.
The fielded system is a different animal, and the difference is usually not documented. Between the design and the running service there are emergency fixes, vendor defaults, configuration drift, a feature flag somebody left on, three years of people adding log lines while chasing problems at two in the morning. None of that is malice or incompetence. It is what operating software looks like.
The system as built is almost never the system as drawn, and a record ends up where that process left it rather than where the architecture document says it should be.
Production systems do not run at one logging level forever.
When a defect reproduces only in production -- and the interesting ones always do -- the standard remedy is to raise log verbosity, gather what is needed, and turn it back down. This is not exotic. It is Tuesday.
Two properties of that practice matter enormously and are almost never thought about together.
First, verbosity is raised per component, not per symptom. You turn up DEBUG on the telephony service, or the scoring service, or the ingest pipeline. That component then logs everything it does, for every request it handles, for the whole window. The ticket that prompted it constrains what anybody looked at. It does not constrain what was captured.
Second, the change is usually driven by a ticket, and the ticket outlives the log. Application logs roll in days or weeks. Ticketing systems are business records kept for years, backed up, owned by a different team. So even when the logs are long gone, the record that verbosity was elevated during a specific dated window is still sitting there.
And verbose logs get moved. Engineers attach excerpts to the ticket, paste them into chat, send them to the vendor for support analysis. Each of those is a durable copy created outside the retention schedule entirely.
This is the reasonable objection, and it is where the problem turns interesting rather than merely inconvenient.
A redaction control is a classifier. It catches what somebody decided was sensitive on the day they wrote the rule. Everything outside that definition passes through untouched.
Now consider what a modern system actually handles. A feature vector. An embedding. A model confidence score. A tokenized representation of something that was once a person's voice or face or payment history. These are float arrays. They do not resemble a name, an email address or a government identifier, and no engineer writing a logging schema three years ago put "embedding" on a PII list -- because at the time, to a reasonable person, it was not obviously identifying.
The control is least likely to cover exactly the category that turns out to matter.
There is also a tension that does not resolve. Verbose logging exists to capture more. Redaction exists to capture less. The same team tunes both, and redacting aggressively enough to be certain destroys the diagnostic value that was the entire reason for raising verbosity. Debugging is the emergency; compliance is the abstraction. Nobody is being cynical. The tool is doing its job.
Here is the shape of the thing.
To say confidently that a system did not retain X, you need to know what it logged.
To know what it logged, you need to know what each logging level emits -- for the specific version of the code that was running. Logging statements are added by whoever is debugging that week. Almost no organization maintains a catalogue reading "at DEBUG, v4.2.1 emits the feature vector."
To know which version was running, you need build and deployment records tying a specific artifact to a specific host at a specific hour. Rolling deploys, canaries and per-region lag make that harder than it sounds, even when everyone is acting in good faith and keeping records.
And if the answer is "redaction would have caught it," then the same two questions re-ask themselves about the redactor: what did its configuration contain, on that date, for that version, and was it applied to the adjacent components or only the one in front of us?
Each attestation depends on another attestation. That is the recursion. Most organizations can enter it and cannot get out, not because anything is wrong with them, but because nobody ever asked them to.
It is tempting to read all of this as being about disputes. It is not, or not primarily.
An organization that cannot answer "what did this system record, on this date, under this configuration" has a problem long before anyone sends a preservation letter. It cannot answer its own regulator. It cannot scope its own breach. It cannot tell a customer what happened to their data. It cannot make a truthful representation in a privacy notice and know it is true next quarter.
The inability is the issue. Who is asking is a detail.
None of this is expensive if it is built rather than reconstructed. Four things, in rough order of value:
Make verbosity changes first-class events. If a level is raised in production, that should generate a durable record: which component, which window, who authorized it, which ticket. Most organizations already have the ticket. Very few treat it as a retention event, which is what it is.
Catalogue what each level emits, per component, per release. Not prose. A test that fails when a new log statement writes a field outside the approved set. A policy is a hope; a build-time check is a control.
Keep deployment records that answer "what was running here, then." Most teams have this data scattered across a CI system, a registry and a config store. Almost none can produce a single answer for an arbitrary past hour.
Classify derived representations explicitly. Decide, in writing, whether an embedding, a feature vector or a score is sensitive for logging purposes -- and decide it before someone else decides it for you. The hardest category is the one that does not look like data about a person until you consider what it can be used to do.
An organization that does all four can answer the question. It may not like its own answer, but it will have one, and it will be true.
An organization that does none of them does not have an answer. It has a diagram, and a sincere belief, and a gap between them that nobody has measured. That is a far worse position to be in -- not because the truth is necessarily bad, but because you find out what it is at the same moment as everybody else.