Deep dive — scenario walkthrough
The sample scenario — what the case-intake AI was doing and why it failed silently
Mid-market plaintiff PI firm (~$500M-$2B recovered / 5,000-20,000 active matters annually), running an AI case-intake triage classifier at the front door. Standard-of-market AI deployment. Standard case-management-platform instrumentation. Standard silent failure.
The AI system
Model: case-intake triage classifier (mock version case-intake-triage-v2.4.7).
Function: at intake, each incoming FNOL / claim inquiry's case type + injury severity + estimated value + client segment is scored 0-100 for complexity + risk. Score routes the matter to one of three lanes:
- Fast-Track Settle (score 0-33) — quick-settle path, junior attorney handles
- Standard Adjuster Review (score 34-66) — typical PI matter, senior attorney
- SIU / Fraud Review (score 67-100) — suspected-fraud / high-complexity, SIU + partner
Training window: client data through 2026-03-15. Deployed Day 1 of the audit period. Not retrained during the 90-day production run.
The client segments
The audit used five common PI intake client segments:
| Client segment | Definition |
|---|
| C1 | Young adult, low income |
| C2 | Young-to-mid adult, upper-middle income |
| C3 | Age 40-55, mid income (the affected segment) |
| C4 | Age 55-70, mixed income |
| C5 | Age 70+, fixed / retirement income |
What shifted during the 90 days
Two input distributions shifted silently and simultaneously:
- Client-mix shift. Marketing outreach amplified C3 representation. C3 share of intake volume grew ~30% post-Day 45.
- Case-type + value shift. Post-shift C3 clients skewed toward medical-malpractice + product-liability + higher estimated case values + older cases (longer days-since-incident). Exactly the pattern where a classifier trained on the historical mix over-scores complexity + risk into the SIU/fraud-review range.
The AI was not retrained. Its baked-in C3-favorable offset from historical training compounded with the shifted-input mix. C3 SIU/fraud-review lane rate rose from 18% baseline to 71% recent. A 53 percentage-point differential, in the wrong direction, silent to every aggregate metric the firm was watching.
What the firm's dashboards showed (green)
- Intake throughput — stable
- Mean triage score — stable
- Aggregate SIU/fraud-review lane count — stable (C1/C2/C4/C5 held, C3 rose, aggregate rounded to noise)
- Model confidence — stable
- Days-to-first-attorney-touch — stable
Standard case-management instrumentation cannot see per-segment differential in aggregate. That is not a criticism of the operational team; it is a structural property of aggregation.
What the independent-verifier sensor saw
- C5 score distribution shape diverged from baseline (KL-divergence 0.71) starting Day 37
- C1 + C4 + C5 all fired high-severity divergence on Day 51 (KL up to 2.25)
- C3 divergence emerged Day 86 alongside the sustained upstream shifts on other segments
- 16 total drift events, 7 high-severity, 9 medium-severity
See how the drift-detection chart is read →
The one sentence the firm's own defense counsel would care about
The AI case-intake triage engine's SIU/fraud-review lane rate for client segment C3 (age 40-55, mid income) rose from 18% to 71% over the audit period — a 53 percentage-point segment-differential in the wrong direction that occurred silently while the firm's case-management dashboard showed all-green throughout.
$499 Snapshot on your firm's actual AI system.
Same scenario structure. Your data, your AI, your jurisdictional footprint. Same 3 business day turnaround.
Buy $499