When we started building Undwrlyft, we had a choice about how to think about accuracy. Standard ML framing optimizes for F1 score — the harmonic mean of precision and recall. Maximize F1, ship the model. But F1 treats false positives and false negatives as equivalent errors, and in legal document review they are categorically not equivalent.
An attorney who runs three reviews and gets three false positive flags — "non-standard indemnity cap" on a clause that is, in fact, market standard — will stop trusting the tool. Not slow their trust. Stop it. The cognitive overhead of checking each flag against their own judgment means the tool is doing negative work: it's creating review steps rather than removing them. We lose that user within the week.
An attorney who gets a false negative — the non-standard clause that we didn't flag — has a worse outcome on a specific deal, but probably doesn't know it happened. The damage is real, but it's distributed and slow.
This is not an argument for ignoring false negatives. Missing a material non-standard clause is an unacceptable error. It's an argument that the path to building a trustworthy legal AI product starts with precision, not recall — and that the product's entire calibration philosophy needs to follow from that.
The Architecture Decision: Three-Stage Extraction
Our clause flagging pipeline has three stages. Understanding the stages explains why we made the accuracy tradeoffs we did.
Stage 1: Clause Boundary Detection
Before we can flag a clause as non-standard, we need to accurately identify where the clause starts and ends in the document. This is harder than it sounds. Legal documents vary enormously in their sectioning conventions — some use numbered sections with clear hierarchy, others use dense paragraph text where multiple clauses are embedded in a single paragraph, and older agreements often have hand-edited formatting that breaks automated segmentation.
We spent a disproportionate amount of time on boundary detection because errors here compound. If we segment a clause incorrectly — capturing part of an indemnification provision but not the limitation-of-liability carve-out in the next sentence — the comparison against the playbook baseline operates on an incomplete unit. The subsequent accuracy analysis becomes meaningless.
Our boundary detection uses a combination of structural signals (section headers, numbered items, paragraph breaks) and semantic signals (does this sentence complete a legal obligation or open one?). For the semantic layer, we use a fine-tuned model trained on annotated legal text, but the structural layer has veto power: if the structural signal and the semantic signal conflict, we defer to structure. Structure in legal documents is intentional; semantic signals in legal prose are more ambiguous than in other text genres.
Stage 2: Clause Classification
Once we have clause boundaries, we classify each clause into one of our covered clause types: indemnification, limitation of liability, change-of-control, IP assignment, termination rights, auto-renewal, governing law, dispute resolution, non-compete and non-solicit, assignment restrictions, confidentiality, and force majeure.
Classification accuracy here is high because the vocabulary of legal clause types is specialized and distinctive. An indemnification clause has strong signal: "shall indemnify," "hold harmless," "defend," combined with an obligation structure. We're not doing sentiment analysis on ambiguous prose; we're pattern-matching on domain-specific legal language that professionals write deliberately.
Where classification is harder: hybrid clauses. A provision that covers both assignment restrictions and change-of-control triggers in a single section. A confidentiality provision that contains an embedded data processing obligation. We classify these by their dominant signal, flag the secondary clause type in the output, and surface both in the flagged brief rather than forcing a single-category label that loses information.
Stage 3: Deviation Assessment
This is where the hard accuracy work happens. We compare the extracted clause against two baselines: the firm's uploaded playbook (if one exists for this clause type) and a market-standard model for this clause type and document type.
The market standard model is not a single position. It's a distribution of positions that we consider "within standard range" for a given clause type in a given document context. An indemnity cap of 1x fees paid is standard in a SaaS MSA. The same cap in an asset purchase agreement is non-standard — most APA indemnity caps run higher, and the deviation in either direction carries different risk profiles depending on whether you're representing the buyer or seller.
The deviation assessment has to be document-type-aware. This is an architectural constraint that isn't obvious from the outside: the same clause text can be standard in one contract type and non-standard in another. We maintain separate baseline distributions by document type, not a single universal baseline.
The Confidence Threshold Decision
Every flagging decision is made against a confidence threshold. We flag a clause as non-standard only when the deviation assessment exceeds that threshold with sufficient confidence. The threshold is calibrated conservatively — we prefer the high-confidence flag over the uncertain flag.
What we don't do: present uncertain flags as if they're certain. If our confidence in a deviation assessment is below the threshold, we include the clause in the "review recommended" category of the output with a lower severity indicator, rather than presenting it as a definitive flag. This gives the attorney visibility into borderline provisions without asserting that borderline is the same as definitively non-standard.
The practical implication: our recall is lower than it could be if we lowered the threshold. We flag fewer things with high confidence rather than many things with low confidence. This is the tradeoff we made deliberately, and it's why our 94% clause anomaly detection rate is a precision-weighted metric, not a raw recall rate. We care about whether the flags are right, not just whether the flags are comprehensive.
Where We're Still Working
We're honest about where the current pipeline has limitations, because overstating accuracy destroys trust just as surely as false positives do.
Heavily negotiated hybrid provisions. In complex deals, major operative provisions have often been negotiated across multiple drafts, and the final text reflects compromise language that doesn't cleanly match any standard form. Our deviation assessment performs less well on these because the baseline distributions weren't trained on negotiated hybrid text. The flag may be technically correct but the severity assessment misses the deal-specific context.
Handwritten amendments and redlines. If a contract has handwritten amendments, or if a Word document has tracked changes that weren't accepted, the text layer we extract may be incomplete or misleading. We flag documents with likely amendment issues for attorney review, but our clause extraction accuracy degrades on these inputs.
Pre-2000 contracts. Older agreements, particularly those that were scanned from physical documents and OCR'd, have degraded text quality that increases boundary detection errors. We process these but with lower confidence, and the output reflects that.
The goal is not to hide these limitations but to surface them clearly in the output so the reviewing attorney knows where to apply additional scrutiny. The flagged brief for a problematic input document should tell the attorney "this document has characteristics that may reduce extraction accuracy" — not silently return a normal-looking brief that happens to be less reliable.
How This Translates to the Product
Every one of these pipeline decisions shows up in the interface. Flags are confidence-weighted, not binary. Document-type context is displayed alongside each flag so the attorney knows whether the standard being applied is the right one for this document. "Review recommended" items are presented separately from high-confidence flags, not mixed together. And for any flag, the attorney can see the exact clause text that triggered the flag — we don't present conclusions without evidence.
That last point matters most: the attorney is always looking at clause text, not at the model's summary of the clause text. The model's job is to surface the right provisions and identify their deviation, not to substitute for the attorney's reading of the contract. The flagged brief is a navigation tool, not a replacement for the review.