Is Law Cooked?
On the November 11 episode of the Moonshots podcast, the hosts went through Anthropic’s new Claude Fable 5.2 and its 16.4 percent score on SemiAnalysis’s new GDPx benchmark, a test of AI agents doing expert work. One host said “remote work is cooked.” Another said “your job will be replaced or can be replaced by an AI definitely by next year.”1 Lawyers do much of their work at a desk, on documents. Is it fair to ask, “Is law cooked?”
So we looked at the part of GDPx the podcast skipped: the legal column. On the benchmark’s own default scoring, every model shown on the legal leaderboard (14 of the 21 tested) scores 0.0 percent. That includes Fable 5.2, Claude Opus 5.5, GPT-6 Astra, and Gemini 4 Carbon.2 SemiAnalysis, which built the test, put it bluntly: “Our finance and legal tasks are so challenging that basically every model scores exactly 0.”3
That doesn’t mean nothing is changing. Some parts of law practice really are being cooked. They just aren’t the parts the headline suggests. Here is what GDPx actually measures, what the models got wrong when given real legal work, and what that means for people who practice law for a living.
1. What GDPx is, and why it is a harder test than the last one
GDPx was released on November 2 by Max Kan, Andrew Megalaa and Dylan Patel of the research firm SemiAnalysis. It has 206 tasks across 11 domains: chip design, cybersecurity, biology, finance, legal, quant trading, recursive self-improvement (AI research), hardware design, materials science, alignment, and software engineering.3,4 By default, an agent gets up to eight hours per task. The headline number is the average of three attempts, and all eleven domains count equally.3 Legal therefore carries one-eleventh of the weight, worth at most about 9.1 points of the headline score (our arithmetic).
The setup is what separates it from earlier benchmarks. OpenAI’s 2025 benchmark, GDPval, gave models a prompt and graded what came back. On that test, Claude Opus 4.1’s outputs were “rated as good as or better than humans in just under half the tasks.”14 SemiAnalysis argues that design “makes GDPval woefully inadequate for evaluating today’s frontier models.”3 GDPx instead puts the agent in a working environment and makes it find what it needs.15
The legal tasks come from Chakra, a lab that builds expert-written tasks for AI training and evaluation.16 According to the GDPx legal page, Chakra’s experts include “corporate partners at Wachtell and other large US law firms,” startup general counsels, and Fortune 500 commercial counsel. The 20 legal tasks cover eight practice areas: M&A, venture capital, capital markets, corporate governance, intellectual property, data privacy, administrative law, and dispute resolution.6 Each task folder looks like a real file: agreements, term sheets, offering documents, court filings, and emails from the client or a partner.6
2. The legal scores: zero on pass/fail, about a third on partial credit
On the default metric, the legal leaderboard is a column of zeros. All 14 models shown score 0.0 percent.2 That is the intended grading for these tasks: a deliverable either meets the bar or it doesn’t. SemiAnalysis gives the same reasoning for its finance tasks: “Messing up a single key cell in an Excel workbook warrants failing the task entirely.”3 A redline that gives away the client’s protective provisions fails the same way.
The site also has a “Partial score” view, which gives partial credit for work graded against the expert-written rubric for each task.5,6 These numbers are the ones that should get a lawyer’s attention:
Bars are scaled to a 50% maximum. On the default pass/fail metric, every model above scores 0.0%.
Three things stand out.
No model is close to finished work. Even the leader’s partial-credit score is 61.6 points short of full marks (our arithmetic).5 For a deliverable a partner would sign, that isn’t a near miss.
The newest model improved, but not dramatically. Fable 5.2’s overall GDPx score nearly doubled over Fable 5.1, from 8.5 to 16.4 percent.4 On the legal partial score it rose 4.6 points, from 29.2 to 33.8 percent (our arithmetic).5 According to one summary of Dylan Patel’s post, he read the release the same way: better at chip design, software, biology and materials, no better at finance, legal or trading.17
The capability is getting cheap fast. OpenAI’s GPT 6.1 Sol earns 30.8 percent partial credit on the legal tasks for $0.73 a task. Fable 5.2 gets 33.8 percent for $10.71, about 15 times the cost for 3 more points (our arithmetic).5 Anthropic lists Fable 5.2 at “$8 per million input tokens and $40 per million output tokens.”18 A first pass that earns about a third of full credit is now very cheap. That is where the economic pressure will hit.
3. Inside the one public legal task: a Series B redline
Most GDPx tasks are private so labs can’t train to the test. Usually only one task per partner is public.15 Chakra’s is quillon-series-b-redline. It’s a routine venture-financing assignment. The agent represents the lead investor in a Series B and gets the company’s draft amended and restated certificate of incorporation, the signed term sheet, the pro forma cap table, a voting-agreement excerpt, and the client’s instructions. The instruction: “Mark it up for Harrow in Word with tracked changes.”19
Here is how it went, according to the GDPx legal page:
- “In 20 rollouts across four models, no redline was usable.” The four were Opus 5.5, GPT-6 Astra, Gemini 4 Argon and GLM 5.3. The best single run met 57 percent of the rubric (Opus 5.5).6
- “20 of 20 runs failed to protect against the company using its subsidiary to bypass the protective provisions of the charter.”6
- 13 of 20 runs didn’t notice from the cap table that the other shareholders had enough votes to strip the client’s protections.6
- 14 of 20 runs applied the client’s emailed provisions without flagging that they re-traded terms already agreed in the signed term sheet.6
The task authors identified three underlying causes: failure to synthesize different input documents, “failure to anticipate how the counterparty could work around the client’s desired protections,” and “treating the client’s email as overriding the signed term sheet.”6
Practicing lawyers will recognize all three. They are the skills that make someone a lawyer rather than a fast typist: reading the whole file, picturing what the other side will do with the language you leave in, and telling a client politely that the email they just sent would reopen a deal that’s already signed. The models produced plausible markups. They didn’t yet do the adversarial thinking the task required.
4. In fairness to the machines (and to the benchmark)
These results shouldn’t be over-read in either direction.
- The grader is a model. In law and hardware design, “language models grade the results using rubrics instead of human experts,” and because most tasks are private, outsiders can’t fully check the difficulty or the grading.15 Chakra says it measures its judge against expert labels and grades each output five times.6
- Twenty tasks, three attempts each, is a small sample, and the work is mostly transactional.6,3 There is no jury charge, no deposition outline, no Georgia premises-liability brief. A zero on a Series B charter tells you little about trial work in either direction.
- The test is designed to be hard. SemiAnalysis wanted a benchmark that frontier models fail most of the time. It wrote at launch that “even the best model scores just 12%,” before Fable 5.2 took that to 16.4 percent.3,4 It also plans to “release new versions every 3-6 months.”3 One Moonshots host predicted GDPx “will be saturated within 6 months.”1 Maybe. The legal column is where that prediction can be tested.
- Vendors report real gains. In Anthropic’s launch post, Harvey’s head of AI research says Fable 5.2 gave clearer direction in an I-9 compliance review and calls it “an exciting improvement in the model’s ability to give practical legal guidance.”18 That’s a customer testimonial, not an independent test, but it fits the partial-credit trend.
5. What is actually getting cooked: the first draft and the hours billed for it
The zero should not reassure anyone, because legal work isn’t paid on a pass/fail basis. Much of it is billed by the hour, and many of those hours go into first drafts, document review, term extraction, and research memos. That kind of work is where a model earning a third of full credit for under a dollar matters most.
The ethics rules make the effect direct. ABA Formal Opinion 512 (2024) says that “lawyers who bill clients an hourly rate for time spent on a matter must bill for their actual time.” The opinion’s example: a lawyer who spends 15 minutes entering information into a tool that drafts a pleading may bill those 15 minutes plus the time spent reviewing the draft for accuracy and completeness. The lawyer can’t bill the hours the draft would have taken by hand.7 For hourly work, the time AI saves comes off the bill.
Clients are noticing. In Thomson Reuters’ Future of Professionals Report 2026 (1,816 professionals in law, tax, audit and related fields), 78 percent of corporate clients called AI-enabled quality improvements very important or essential, but only 6 percent said most or all of their providers deliver them. Thirty-two percent said they have reconsidered, or plan within 12 months to reconsider, firms they see as falling behind. Thomson Reuters estimates about $143 billion in U.S. legal and CPA client revenue is “under active reconsideration.”8 Thomson Reuters sells legal AI, so read those figures with that in mind. The survey still shows the direction of client pressure.
So the honest answer to “is law cooked?” is that the hourly first draft is cooked. Judgment and accountability are not.
6. What isn’t getting cooked: someone has to be accountable
A wrong cell can wreck a financial model. In law, a wrong citation can bring sanctions, a bar complaint, or a client who loses a case that should have been won. Courts are now documenting this kind of failure regularly.
Researcher Damien Charlotin’s AI Hallucination Cases database tracks court decisions where a party relied on hallucinated content. It lists 2,217 so far.9 Georgia has its own example. In Shahid v. Esaam (Ga. Ct. App. June 30, 2025), a trial court denied a wife’s petition to reopen a divorce with an order that cited two cases that don’t exist. The order appears to have been prepared by the husband’s lawyer. On appeal, the same lawyer’s brief cited more fake or irrelevant authority, including in support of a request for appellate attorney fees. The Court of Appeals vacated the order and imposed a frivolous-motion penalty, writing: “We are troubled by the citation of bogus cases in the trial court’s order.”10
Specialized legal-research products have not solved this. In a 2024 Stanford RegLab and HAI study, “the Lexis+ AI and Ask Practical Law AI systems produced incorrect information more than 17% of the time, while Westlaw’s AI-Assisted Research hallucinated more than 34% of the time.”20 Those are 2024 products, and newer versions should be retested. The lesson still holds: a citation can be real and still not support the proposition it is cited for.
That is why the ABA’s rule matters more than any benchmark. Opinion 512 warns that “uncritical reliance on content created by a GAI tool can result in inaccurate legal advice to clients or misleading representations to courts and third parties.”7 In Georgia, the Judicial Council’s Ad Hoc Committee on AI and the Courts, chaired by Justice Andrew Pinson, submitted its report in July 2025. A State Bar committee chaired by Darrell Sutton is reviewing how AI affects the Georgia Rules of Professional Conduct.21 Whatever those rules end up saying, the lawyer who signs the filing remains responsible for it.
Employment projections don’t show a collapse either. The Bureau of Labor Statistics projects lawyer employment to grow 5 percent from 2025 to 2035, with about 28,700 openings a year and a median wage of $159,670 in May 2025.22 BLS does expect some routine legal work to be automated, but it says “this increased efficiency is not expected to reduce overall demand for lawyers.”22 Projections can be wrong, and these may understate how quickly AI is improving. But they don’t show an occupation in freefall.
7. The next frontier: AI-run companies, and who gets sued
The same Moonshots episode spent a segment on something more consequential for litigators than any benchmark: Delaware’s draft bill for an “Artificial Intelligence Company,” or AIC.1
Under the draft, the manager is “the AI agent that the member has designated to manage the business and affairs, and to conduct the business, of the AIC.”23 As Ars Technica reports, an AIC “may have any lawful purpose, regardless of whether for profit, except for the business of banking,” and “The member is not liable for any act or omission that the AIC takes or fails to take, or that another person takes or fails to take at the direction of the AIC.” The exceptions cover fraud, willful violations of law, and conduct the member knew or should have known was outside the AIC’s stated purpose.11 The bill proposes a 30-month regulatory sandbox.11
Delaware’s Secretary of State presented the idea in Fortune as accountability: “The answer is to wrap AI in legal form.”24 House Oversight Chairman James Comer sees it differently. He wrote that the proposal “could be used to protect powerful AI companies and their investors from responsibility for reckless harmful actions.”25 According to WHYY, Delaware lawmakers could take up the first-in-the-nation legislation next year.26
For plaintiff lawyers, this is the part to watch. When an AI agent signs a contract, runs a fleet, or manages a property and someone gets hurt, the questions will be familiar: who had the duty, who is the defendant, who is insured, and whether the liability shield holds. Analyzing that is the work GDPx says models still fail at: reading the whole record, anticipating the workaround, and refusing to treat the latest email as if it overrides the signed agreement.
8. What the future holds for practitioners
Based on the evidence above rather than the hype, here is our read:
- Hourly billing for first drafts is going away. Opinion 512 says efficiency gains can’t be billed as hours.7 Clients now have a tool that earns about 30 percent partial credit on expert legal tasks for under a dollar a task.5 Flat fees, contingency work, and value-based pricing are better positioned. That’s good news for contingency-fee plaintiff lawyers, who already get paid for results, not hours.
- Verification becomes a core skill. The lawyers who do well will check every citation and every defined term before filing, and will say so. Charlotin’s database, with more than two thousand decisions involving lawyers, self-represented litigants and others, shows what happens when no one checks.9
- Judgment is what clients pay for. The redline failures were failures of synthesis, anticipating the counterparty, and pushing back on the client.6 Those are what a client pays a lawyer for.
- Training junior lawyers gets harder. 48 percent of professionals surveyed worry about AI’s effect on developing independent judgment, and 71 percent say early-career professionals need structured support from experienced colleagues.8 Junior lawyers learned judgment by doing the first drafts that models now produce. Firms will need a deliberate replacement.
- Watch the legal column, not the headline. The headline will keep going up as labs improve at chip design and software. The signal for lawyers is the day the legal pass/fail column moves above zero. With new GDPx versions planned every three to six months, we’ll be checking.3
9. How would we know? Measuring AI progress in personal-injury law
GDPx’s 20 legal tasks span eight corporate and transactional practice areas, from M&A and venture capital to dispute resolution, and they were developed with an expert roster of large-firm corporate partners, startup general counsels and in-house counsel.6 Personal injury isn’t one of the areas. A plaintiff’s file looks different: a box of medical records, a crash report, a recorded statement, a time-limited demand, a summary-judgment fight over what a store manager knew about a puddle. So the zero above says little about personal-injury practice in either direction. The more useful question for this side of the bar is how lawyers and AI labs could tell, in a way someone else can check, whether AI is actually getting better at this work.
The clearest framework we have found comes from Justin Curl, who holds a law degree from Harvard and a computer-science degree from Princeton and has written widely on AI and the practice of law. One disclosure: his site now lists him as a Member of Technical Staff at Anthropic, the company behind the Fable and Opus models discussed above.27 In “AI Won’t Automatically Make Legal Services Cheaper” (Lawfare Research Paper Series, 2026), Curl, Sayash Kapoor and Arvind Narayanan separate a lawyer’s inputs (hours, tools), outputs (motions, contracts, briefs) and outcomes. “Outcomes are what clients actually care about: disputes resolved, deals closed, and rights protected.”12 They also describe legal services as “credence goods,” whose quality is hard to judge even with hindsight.12 Those two ideas define the measurement problem. A tool can multiply the demand letters a firm sends (outputs) without getting a single client a better recovery (outcomes), and the client usually can’t tell which happened.
Why the familiar scoreboards don’t settle it
Exams. GPT-4’s reported 90th-percentile bar-exam score was the first big piece of evidence offered for the idea that AI was coming for lawyers. Eric Martínez’s re-analysis in Artificial Intelligence and Law estimated that, compared only with test-takers who passed, GPT-4 placed around the 48th percentile overall and around the 15th percentile on essays.28 In “Promises and Pitfalls of Artificial Intelligence for Legal Applications”, Kapoor, Peter Henderson and Narayanan explain why exams mislead even when scored correctly: the bar exam “overemphasizes precisely the thing that language models are good at,” which is recall of subject-matter knowledge, and contamination (test material leaking into training data) can inflate scores.13 Curl, Kapoor and Narayanan build on that critique in the 2026 report.12 The 2024 paper’s central finding belongs in every legal-tech purchasing meeting: “Tasks that would lead to the most significant changes to the legal profession are also the ones most prone to overoptimism about AI capabilities, as they are harder to evaluate.”13
Outcome prediction. Predicting how a case will come out sounds like the most checkable test there is. But the same paper describes a review of 171 studies claiming to predict court decisions and reports that “only 12 of 171 papers (7%) end up predicting court decisions.” Many of the rest fed the model the text of the judgment, which already contains the answer.13
Model graders. GDPx uses language models, not human experts, to grade the legal work against rubrics.15 So does the Vals Legal AI Report, which calls that approach “becoming a standard in evaluation for generative AI settings.”29 A model grader is fine if someone checks it against experts. OpenAI’s HealthBench shows how. It was built with 262 physicians who have practiced in 60 countries, covers 5,000 health conversations, and grades against 48,562 physician-written rubric criteria. OpenAI then measured how often the model grader agreed with physicians: “We found similar pairwise agreement between models and physicians as between individual physicians.”30 We could not find anything comparable for personal-injury work.
What a verifiable personal-injury test would look like
Kapoor, Henderson and Narayanan’s first recommendation is to “Improve construct validity by involving legal experts in evaluation.”13 Lawyers and labs have done this before. LegalBench assembled 162 tasks, many “designed and hand-crafted by legal professionals,”31 and the GDPx legal tasks come from Chakra experts who include large-firm partners and in-house counsel.6 Here is how the two sides could divide the work for personal injury.
- Lawyers from both sides write the tasks and the rubrics. Plaintiff lawyers, defense lawyers, claims professionals and retired judges should each grade for what their side would catch. Good candidate tasks: a medical chronology built from thousands of pages of records, a time-limited demand, a deposition outline for a store manager, a response to a premises-liability summary-judgment motion, and an apportionment analysis. The Vals study already tested one of these. On chronology generation, the best AI tool matched a lawyer control group at 80.2 percent but did not beat it.29 Labs bring what most firms can’t: the infrastructure to run every task many times, private held-out tasks like most of GDPx’s,15 and checks of the model grader against expert judgment like HealthBench’s.30
- Let the law grade what it can. Some personal-injury work has a checklist written into the statute. Under the current text of O.C.G.A. § 9-11-67.1, rewritten effective July 1, 2024, a pre-answer offer to settle a motor-vehicle injury claim that a lawyer helps prepare “shall contain the following material terms, which shall be the only material terms.” There are seven: an acceptance date at least 30 days after receipt, the amount, the parties to be released, the type of release with an itemization of what each releasee gets, the claims to be released, a payment date at least 40 days after receipt, and a requirement that the recipient provide “a statement, under oath,” about its insurance coverage, which the offeror may waive. The offer must also include medical or other records “sufficient to allow the recipient to evaluate the claim,” go by certified mail or statutory overnight delivery, and “specifically reference this Code section.”32 A model’s demand either complies or it doesn’t, the same pass/fail standard GDPx uses. The same statute grades the defense side too. Under subsection (i), an insurer that accepts the material terms in writing, provides any required sworn coverage statement, and pays the lesser of the demand or its bodily-injury limits by the dates in the offer cannot be sued for failing to settle.32 Getting the checklist wrong has real consequences. In Lester v. Hampton (Ga. Ct. App. Oct. 23, 2025), decided under the earlier five-term version, a pre-answer demand named an amount and a deadline but not the parties to be released, the type of release or the claims to be released. The Court of Appeals held that it “was not a valid offer capable of being accepted” and reversed a trial court that had enforced a $25,000 settlement.33
- Tag every task with the law in effect on its date. The checklist itself changes. Lester applied the 2021 version of the statute, noting in a footnote that it “has since been amended.”33 In Torres v. Pineda (Ga. Ct. App. Jan. 27, 2026), the court applied the 2024 version and described the material terms as enumerated in subsection (b)(1), including the sworn statement about insurance coverage under (b)(1)(G).34 Even the 2024 version’s start date is a trap: the act took effect July 1, 2024, but its uncodified applicability section reaches offers “made on or after April 22, 2024,” the day the Governor signed it.32 Georgia’s 2025 tort-reform law then changed the rules on medical-damages evidence, “anchoring” pain-and-suffering arguments, negligent-security liability, and seatbelt evidence.35 A test that doesn’t tag each task with the law in effect on a given date will reward a model for reciting a superseded rule.
- Grade against reality, before the answer exists. The cleanest defense against contamination is to ask for predictions about events that haven’t happened yet. This site’s premises summary-judgment survey counts 44 published Court of Appeals opinions, filed October 1, 2023 through September 30, 2026, that decided a summary-judgment ruling in a premises-liability case. Twenty-nine of them (66 percent) favored the landowner.36 A model that always picked the landowner would have been right about two times in three on that set (our arithmetic), so a forecaster has to beat that base rate before it shows any legal judgment. Logging a model’s prediction before each new opinion issues, and scoring it when the opinion comes out, would produce a test no lab could have trained on.
- Run randomized trials with real work. Legal scholars have already shown how. In the first randomized trial of AI help on legal tasks, Jonathan Choi, Amy Monahan and Daniel Schwarcz randomly assigned sixty University of Minnesota law students (fifty-nine finished) to draft a complaint, a contract, a section of an employee handbook and a client memo, with each student using GPT-4 on two of the four tasks. The authors graded every submission “anonymously, with no knowledge of participant identity or GPA, GPT use, or time spent on task,” and “pre-registered our methods and hypotheses prior to analyzing our results.”37 GPT-4 “only slightly and inconsistently improved the quality of participants’ legal analysis but induced large and consistent increases in speed.” Grades moved by +0.17, +0.24, +0.07 and −0.07 on a 4.0 scale, while time on task fell by 24.1, 32.1, 21.1 and 11.8 percent. Only the contract-drafting grade gain was statistically significant at the 95 percent level, and “where it was useful at all, the lowest-skilled participants saw the largest improvements.”37 The task closest to injury work did worst. The client memo asked whether a client should put an allergen warning on a product, a product-liability question. It was the only task where AI-assisted grades went down (by 0.07, not statistically significant), and it had the smallest time savings.37
- Repeat the trials for every new generation of tools. A follow-up trial shows why one study is never enough. In “AI-Powered Lawyering,” published in 2026 in the Journal of Law & Empirical Analysis, Daniel Schwarcz, Sam Manning, J.J. Prescott, Patrick Barry, David R. Cleveland and Beverly Rich tested two late-2024 tools: OpenAI’s o1-preview, the first publicly available reasoning model, and vLex’s Vincent AI, a legal research tool built on retrieval-augmented generation (RAG). Of the Minnesota and Michigan law students who enrolled, 137 completed at least one task and 125 completed all six. The six tasks were junior-associate work developed with practicing attorneys: a client email, a research memo, an analysis of a complaint, a nondisclosure agreement, a motion to consolidate and a persuasive letter. Each student did two with o1-preview, two with Vincent AI and two with no AI. Three co-authors graded blind on accuracy, analysis, organization, clarity and professionalism, each on a 1–7 scale, with a separate flag for hallucinated sources. “All three graders have legal practice experience and were uninvolved in data collection or analysis.” The vendors cooperated without paying: “Neither OpenAI nor VLex (the company that owns Vincent AI) provided financial support for this project, but they both did make their AI platforms freely available to study participants.”38 That is a workable template for how labs and lawyers can split the job.
- Report accuracy on its own line. Averaged over every task and quality measure, access to Vincent AI raised scores by 0.27 points on the seven-point scale and o1-preview by 0.53, which the authors put at roughly 0.15 to 0.20 and 0.30 to 0.35 standard deviations. Most of the gain came in clarity, organization and professionalism, and o1-preview also improved analysis, but “we find no evidence that either tool improves Accuracy, including Vincent AI, the RAG system.” Both tools cut time “by roughly 20–28% on five of the six tasks,” close to the GPT-4 savings in the Choi trial, and the authors report productivity gains “of anywhere from 50% to 130%” on five of six tasks. The exception was the nondisclosure agreement, a template-driven transactional task on which neither tool helped.38 The task closest to an injury practice was the four-hour research memo. The working-paper version prints it in full: a trucking company’s commercial auto liability policy carries a $2 million limit, and the question is whether, under Massachusetts and New Hampshire law, the insurer must also pay $200,000 in attorneys’ fees.39 On that memo, once each student’s own baseline was accounted for, “Neither tool appears to improve performance on the NDA or legal memo tasks,” and o1-preview users’ accuracy scores fell by 0.67 points (22.1 percent), a statistically significant drop. Both tools still cut time on the memo by about a quarter, so the study’s productivity measure, quality points per share of allotted time, rose 80 percent with Vincent AI and 107 percent with o1-preview.38 On a coverage question, a faster memo with the law wrong is not progress. In our view, that is why a personal-injury benchmark should report accuracy separately instead of folding it into a single productivity number.
- Don’t measure progress by how the tools feel. After the trial, students rated Vincent AI more helpful for quality, though o1-preview delivered the broader quality gains. The authors conclude that participants “have somewhat unreliable intuitions about where the tools were most and least helpful.” They also found that the tools “appear to provide the largest gains for participants with lower baseline skill.”38 User-satisfaction surveys and vendor testimonials, in other words, are not evidence of progress. The authors’ own prescription matches this post’s: legal benchmarks “are becoming increasingly inadequate,” partly because many are saturated, and “we believe it is important and timely for clients, law firms, and law schools to start embedding such trials into their operations and adapt accordingly.”38
- Build on those trials, and count the checking time. Choi, Monahan and Schwarcz are candid about what a better trial would fix. Their sample was small, so “our estimates are generally noisier than we would like.” The tasks were simplified; for example, the memo “did not require independent research.” And each task was graded by “a single investigator using a pre-determined grading rubric.” They also note that participants were paid a flat rate, whereas “under time-based client billing, lawyers have an economic incentive to spend as much time as possible on a task in order to maximize revenue,” so the speed gains might not carry over to practice.37 Contingency-fee injury work, which is paid out of the recovery rather than by the hour, avoids that problem; that observation is ours, not the authors’. The follow-up lists its own limits. Its participants “were not fully licensed lawyers but upper-level law students,” and “participants knew when they were permitted to use an AI tool.” Graders, though blind, might still “intuit which tasks were more likely to have been supported by AI.” The authors “did not counterbalance the tasks by varying the order in which participants completed them,” and “None of the six tasks, however, were ones that senior lawyers or law firm partners would typically perform.” By their own description, they “preregistered a rough outline of the experiment” before analyzing results.38 A personal-injury version would use practicing lawyers and paralegals on de-identified files, pre-register its full analysis plan, grade blind with more than one grader per task, vary the task order, and charge the AI for the time it takes to verify its work. In Curl’s interviews with large-firm lawyers, a senior associate said that having a junior colleague double-check an AI analysis “ended up taking more time than he saved from using AI.”40
- Score reliability, not the best run. A lawyer needs a tool that is right every time, not once in eight tries. Researchers at Sierra proposed a metric for this in τ-bench, “pass^k,” defined as “the chance that all k i.i.d. task trials are successful.” On their retail tasks, even agents built on GPT-4o fell below 25 percent at pass^8.41 GDPx reports the average of three attempts.3 The Series B redline shows why the difference matters: the best single run met 57 percent of the rubric, and none of the 20 runs was usable.6
- Audit the products, then track outcomes. Benchmarks test models, but firms buy products. In “A Blueprint for Auditing Generative AI,” Jakob Mökander, Curl and Mihir Kshirsagar propose three layers: governance audits of the companies, model audits before release, and application audits of the products built on top.42 For a firm buying a demand-letter or chronology tool, the application audit is the one that matters. Over time, the measure should shift from outputs to outcomes. Curl, Kapoor and Narayanan argue that “regulatory markets may be able to assess AI’s utility for legal services more reliably than benchmarking,” and they discuss Utah’s and Arizona’s regulatory sandboxes.12 In “To Keep Human Judgment, We Need AI Courts,” Curl proposes a court-run AI adjudication track that would amount to “effectively A/B testing AI and human judges,” with appeal and reversal rates as the quality measures: “low reversal rates would be excellent evidence of the quality of AI decisions.”43
One constraint shapes all of this. Personal-injury files are full of medical records and privileged communications, so a shared test can’t simply be uploaded to a lab. The law on AI and privilege is unsettled. In United States v. Heppner, Judge Jed Rakoff found that documents a criminal defendant created using Claude were not protected by attorney-client privilege or the work-product doctrine. Curl and Kshirsagar agree with the result but criticize the reasoning, which relied on the AI company’s terms of service.44 Until that is settled, any shared personal-injury benchmark should be built from synthetic or properly de-identified files used with client consent.
A scorecard: what would count as verified progress
These are our proposed tests, not results anyone has reported yet. We would count AI as making verified progress in personal-injury work when:
- Pass/fail scores on tasks the law can grade, such as a pre-answer demand that meets every requirement of § 9-11-67.132 or a damages argument that follows the 2025 tort-reform rules,35 rise above zero and stay there across repeated runs, measured the way pass^k measures it.41
- Under blind grading by both plaintiff and defense lawyers, AI-assisted work beats lawyer-only work on chronologies, demands and summary-judgment responses, with checking time charged to the AI. For comparison, the best tool in the Vals study only matched lawyers on chronologies.29,40
- Forecasts of Georgia appellate premises decisions, logged before the opinions issue, beat the two-in-three base rate.36
- Court decisions involving AI-fabricated citations start declining even as more lawyers use AI. Charlotin’s database records those decisions, so the first half of that trend can be checked.9
- Client outcomes improve, not just lawyer outputs: faster resolutions and better net recoveries for comparable injuries. That is the hardest to measure and the one that matters most.12
10. What AI still has to get better at
Good measurement shows the gap. Closing it takes advances in the models and in the tools built around them. The research above points to seven.
- Citations that are grounded and checkable. Testing ChatGPT 4, ChatGPT 3.5, PaLM 2 and Llama 2, Matthew Dahl, Varun Magesh, Mirac Suzgun and Daniel E. Ho found in “Large Legal Fictions” that, asked a direct, verifiable question about a randomly selected federal case, the models hallucinated “between 58% (ChatGPT 4) and 88% (Llama 2) of the time” and “often uncritically accept users’ incorrect legal assumptions.”45 Purpose-built research tools did better, but in a follow-up Stanford study Lexis+ AI and Ask Practical Law AI still gave incorrect information more than 17 percent of the time, and Westlaw’s AI-Assisted Research hallucinated more than 34 percent of the time.20 Grounding helps, but only partly. In the Schwarcz and Manning trial, graders found hallucinated sources in 3 submissions made with the retrieval-based Vincent AI, 4 made with no AI and 11 made with the o1-preview reasoning model, yet Vincent AI did not improve accuracy scores. The authors trace that to retrieval tools’ “inability to reliably identify and leverage the most relevant sources among the millions of potentially relevant cases, statutes, regulations, and secondary materials.” They suggest that “integrating domain-specific RAG capabilities with reasoning models could yield even larger improvements.”38 That combination, tested on Georgia injury files, is an obvious next trial. The standard to aim for is a model that cites only authority that supports the proposition and shows the passage that does.
- Knowing what it doesn’t know. Dahl and his co-authors found that models “struggle to accurately gauge their own level of certainty without post-hoc recalibration.”45 Even in medicine, OpenAI reports substantial room for improvement in “seeking necessary context for underspecified queries and worst-case reliability.”30 A personal-injury intake is underspecified by nature. A useful model asks about prior injuries and gaps in treatment before it values a claim.
- Local, current law. The same study found models perform better on cases that are “more salient, and from more prominent legal jurisdictions.”45 Georgia premises and auto practice is close to the opposite: state-court law, statutes amended in 2024 and 2025, and a body of Court of Appeals opinions that rarely makes national news.34,35 People get this wrong too. In Lester, the Court of Appeals noted that the trial court had relied on a 2015 decision about a 2011 collision, which arose before the settlement-offer statute applied.33
- Reading the whole file and anticipating the other side. The GDPx redline failures came from not synthesizing the documents and not anticipating the counterparty’s workaround.6 Litigation raises the stakes. One partner told Curl that “corporate legal work is more of a good-enough practice than a perfection practice because no one is trying to ruin your life.”40 In a personal-injury case, someone is: the defense will read every record the model summarized.
- Consistency over long tasks. pass^k shows how quickly reliability falls as attempts are repeated.41 Curl cites METR’s finding that the length of tasks AI agents can reliably complete has been doubling roughly every seven months.40 METR’s measurements come from “over a hundred diverse software tasks,”46 though, and a lawsuit runs for months or years. Nobody yet publishes a comparable time-horizon measure for legal work.
- Making verification cheap. Curl’s interviewees used AI most where checking was easy. As one associate put it, “AI is great for the first and last pass at things.” Curl’s point about pinpoint source links applies directly to medical chronologies: “Features like these don’t make AI tools any more capable. But by making verification faster, they let users capture more of the productivity gains.”40
- Independence and confidentiality. Curl, Peter Henderson, Kart Kandula and Faiz Surani argue that judges shouldn’t rely on AI for the ordinary meaning of text, in part because language models are “inherently shaped by private interests.”47 Lawyers relying on a model to value a case or summarize a record need to know whose judgment is built into it. And they need deployment terms that keep client files confidential.44
Even a perfect model would run into what Curl, Kapoor and Narayanan call the three bottlenecks: “regulatory barriers, adversarial dynamics, and human involvement.”12 Personal injury is adversarial by design, and insurers will have the same tools plaintiffs do. When both sides get more productive, the authors warn, “parties are locked into an arms race of increasing amounts of legal work required to reach the same outcome.”12 Courts are already absorbing the volume. Curl cites research by economists Anand Shah and Joshua Levy finding that the share of federal complaints containing AI-generated text rose from about 0.1 percent before ChatGPT to 18 percent by early 2026.43 Contingency practice may be the best place to test whether AI improves outcomes rather than outputs. As Curl notes, under contingency fees “Lawyers are paid when their clients achieve their desired legal outcome.”40 The fee is already tied to the measure that counts.
The bottom line
On the benchmark behind the podcast’s “remote work is cooked” talk,1 no model shown passed a single legal task under the default grading, and on the one public legal task, four leading models missed a subsidiary workaround in all 20 runs.2,6 That doesn’t mean law is safe. It means the threat is economic before it is existential. The cheap, fast first draft that earns about a third of full credit is here and will cut into hourly billing. The lawyer who reads the whole file, anticipates the other side, and answers for what gets filed is still needed.
The argument over whether AI can practice law won’t be settled by the next headline benchmark. It can be settled by a shared test of real personal-injury work: written by lawyers on both sides, versioned to the law in effect, graded by the statute where it can be and by blind expert review where it can’t, run enough times to measure reliability, and finally judged by what happens to clients.12,13 Labs have the infrastructure to build it, and lawyers have the files and the judgment. Neither can do it alone.
Sources
- Peter H. Diamandis et al., “Jensen’s Spy Probe, Anthropic’s Debut Week, and Elon’s 30M Robot Workforce,” Moonshots #310 (YouTube, Nov. 11, 2026). Read the source (youtube.com)
- SemiAnalysis, GDPx Leaderboard, Legal domain (Chakra, 20 tasks), default “Score” metric (viewed Nov. 12, 2026). Read the source (gdpx.semianalysis.com)
- Max Kan, Andrew Megalaa & Dylan Patel, “GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains,” SemiAnalysis (Nov. 2, 2026). Read the source (newsletter.semianalysis.com)
- SemiAnalysis, “GDPx: The General Agentic Benchmark,” overall leaderboard (viewed Nov. 12, 2026). Read the source (gdpx.semianalysis.com)
- SemiAnalysis, GDPx Leaderboard, Legal domain (Chakra, 20 tasks), “Partial” metric (viewed Nov. 12, 2026). Read the source (gdpx.semianalysis.com)
- Chakra (with SemiAnalysis), “Legal: Chakra,” GDPx domain page. Read the source (gdpx.semianalysis.com)
- ABA Standing Comm. on Ethics & Prof’l Responsibility, Formal Opinion 512, Generative Artificial Intelligence Tools (July 29, 2024). Read the source (lawnext.com)
- Bob Ambrogi, “Thomson Reuters ‘Future of Professionals’ Report Warns of Widening Gap Between AI Adoption and AI Value,” LawSites (June 2026). Read the source (lawnext.com)
- Damien Charlotin, “AI Hallucination Cases” database (viewed Nov. 12, 2026). Read the source (damiencharlotin.com)
- Eugene Volokh, “Georgia Trial Court Cites Likely AI-Hallucinated Cases (Possibly Borrowed from Party’s Filing),” The Volokh Conspiracy (July 3, 2025) (quoting Shahid v. Esaam, No. A25A0196 (Ga. Ct. App. June 30, 2025)). Read the source (reason.com)
- Jon Brodkin, “‘No human at the helm’: Delaware plan would let AI agents run entire businesses,” Ars Technica (Oct. 2026). Read the source (arstechnica.com)
- Justin Curl, Sayash Kapoor & Arvind Narayanan, “AI Won’t Automatically Make Legal Services Cheaper,” AI as Normal Technology (Feb. 12, 2026) (also published in Lawfare’s Research Paper Series). Read the source (normaltech.ai)
- Sayash Kapoor, Peter Henderson & Arvind Narayanan, “Promises and Pitfalls of Artificial Intelligence for Legal Applications,” Journal of Cross-disciplinary Research in Computational Law (2024) (arXiv:2402.01656). Read the source (arxiv.org)
- OpenAI, “Measuring the performance of our models on real-world tasks” (GDPval) (Sept. 25, 2025). Read the source (openai.com)
- “No AI agent scores above 12 percent on SemiAnalysis’ new real-world work benchmark,” The Decoder (Nov. 2026). Read the source (the-decoder.com)
- Chakra, “Task factories for useful intelligence”. Read the source (chakra.dev)
- MindPattern, “Dylan Patel says GDPx shows Fable 5.2 got better at chip design, software engineering, biology and materials, and no better at finance, legal or trading” (Nov. 11, 2026). Read the source (mindpattern.ai)
- Anthropic, “Introducing Claude Fable 5.2 and Claude Mythos 5.2” (Nov. 9, 2026). Read the source (anthropic.com)
- GDPx, “quillon-series-b-redline: rollouts” (public legal task). Read the source (gdpx.semianalysis.com)
- Stanford HAI, “AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries” (May 23, 2024). Read the source (hai.stanford.edu)
- Supreme Court of Georgia, “Judicial Council AI Committee Submits Report” (July 3, 2025). Read the source (gasupreme.us)
- U.S. Bureau of Labor Statistics, Occupational Outlook Handbook, “Lawyers”. Read the source (bls.gov)
- Delaware draft House Bill, “An Act to Amend Title 6 of the Delaware Code Relating to Artificial Intelligence Companies” (draft of June 2, 2026, as published by Ars Technica). Read the source (cdn.arstechnica.net)
- Charuni Patibanda-Sanchez et al., “Exclusive: Delaware proposes testing the AIC, a new legal entity for agents in a regulatory sandbox,” Fortune (July 14, 2026). Read the source (fortune.com)
- House Committee on Oversight and Government Reform, “Comer Investigates Potential for Harm, Lack of Transparency in Delaware Initiative to Create New Artificial Intelligence Companies” (Oct. 2026). Read the source (oversight.house.gov)
- WHYY, “Delaware could be first state to allow AI-run companies” (updated Sept. 30, 2026). Read the source (whyy.org)
- Justin Curl, personal site and publication list (viewed Nov. 12, 2026). Read the source (justincurl.github.io)
- Eric Martínez, “Re-evaluating GPT-4’s Bar Exam Performance,” Artificial Intelligence and Law (2024) (open access). Read the source (link.springer.com)
- Vals AI & Legaltech Hub, “Vals Legal AI Report” (Feb. 27, 2025). Read the source (vals.ai)
- OpenAI, “Introducing HealthBench: An evaluation for AI systems and human health” (May 12, 2025). Read the source (openai.com)
- Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré et al., “LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models” (2023) (arXiv:2308.11462). Read the source (arxiv.org)
- O.C.G.A. § 9-11-67.1 (2025), as rewritten by Ga. L. 2024, p. 178 (SB 83), eff. July 1, 2024 (Justia text with Code Commission and editor’s notes; checked against Westlaw, current through the 2026 Regular and First Special Sessions). Read the source (law.justia.com)
- Lester v. Hampton, No. A25A1078 (Ga. Ct. App. Oct. 23, 2025) (full opinion, CourtListener). Read the source (courtlistener.com)
- Cunningham Bounds, “Statutory Offers to Settle under OCGA § 9-11-67.1” (Feb. 2, 2026) (summarizing Torres v. Pineda, No. A25A1634 (Ga. Ct. App. Jan. 27, 2026)). Read the source (cunninghambounds.com)
- Office of the Governor of Georgia, “Gov. Kemp Signs Historic Legislation Delivering Commonsense, Meaningful Tort Reform” (Apr. 21, 2025) (SB 68). Read the source (gov.georgia.gov)
- Slip and Y’all, Premises summary-judgment survey: 44 published Ga. Ct. App. opinions, Oct. 1, 2023 – Sept. 30, 2026. Read the source (slipandyall.com)
- Jonathan H. Choi, Amy B. Monahan & Daniel Schwarcz, “Lawyering in the Age of Artificial Intelligence,” 109 Minnesota Law Review 147 (2024) (full text via the University of Minnesota Law School scholarship repository). Read the source (doi.org)
- Daniel Schwarcz, Sam Manning, J.J. Prescott, Patrick Barry, David R. Cleveland & Beverly Rich, “AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice,” 3 Journal of Law & Empirical Analysis 220 (2026) (open access). Read the source (journals.sagepub.com)
- Schwarcz, Manning, Barry, Cleveland, Prescott & Rich, “AI-Powered Lawyering” (working paper, Mar. 2025), Appendix A (full text of the six assignments). Read the source (law.berkeley.edu)
- Justin Curl, “AI is just starting to change the legal profession,” Understanding AI (Jan. 15, 2026). Read the source (understandingai.org)
- Shunyu Yao, Noah Shinn, Pedram Razavi & Karthik Narasimhan (Sierra), “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” (2024) (arXiv:2406.12045). Read the source (arxiv.org)
- Jakob Mökander, Justin Curl & Mihir Kshirsagar, “A Blueprint for Auditing Generative AI,” in Research Handbook on Law of Artificial Intelligence (2d ed. 2025) (arXiv:2407.05338). Read the source (arxiv.org)
- Justin Curl, “To Keep Human Judgment, We Need AI Courts,” Pax Machina (Oct. 12, 2026). Read the source (paxmachina.ai)
- Justin Curl & Mihir Kshirsagar, “AI and Privilege After United States v. Heppner,” Lawfare (2026). Read the source (lawfaremedia.org)
- Matthew Dahl, Varun Magesh, Mirac Suzgun & Daniel E. Ho, “Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models,” Journal of Legal Analysis (2024) (arXiv:2401.01301). Read the source (arxiv.org)
- METR, “Task-Completion Time Horizons of Frontier AI Models” (last updated Sept. 8, 2026). Read the source (metr.org)
- Justin Curl, Peter Henderson, Kart Kandula & Faiz Surani, “Judges Shouldn’t Rely on AI for the Ordinary Meaning of Text,” Lawfare (2025). Read the source (lawfaremedia.org)
Further reading
- Magesh et al., “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools” (Stanford RegLab). Link (reglab.stanford.edu)
- ABA News, “ABA issues first ethics guidance on a lawyer’s use of AI tools” (July 29, 2024). Link (americanbar.org)
- Judicial Council of Georgia, Artificial Intelligence and Georgia’s Courts (2025). Link (gasupreme.us)
- Shahid v. Esaam, No. A25A0196 (Ga. Ct. App. June 30, 2025) (full opinion, FindLaw). Link (caselaw.findlaw.com)
- GDPx, public task rollouts for every domain. Link (gdpx.semianalysis.com)