We conducted the first randomized controlled trial to study the effect of AI assistance on human legal analysis. We randomly assigned law school students to complete realistic legal tasks either with or without the assistance of GPT-4. We tracked how long the students took on each task and blind-graded the results. We found that access to GPT-4 only slightly and inconsistently improved the quality of participants’ legal analysis but induced large and consistent increases in speed. AI assistance improved the quality of output unevenly—where it was useful at all, the lowest-skilled participants saw the largest improvements. On the other hand, AI assistance saved participants roughly the same amount of time regardless of their baseline speed. In follow up surveys, participants reported increased satisfaction from using AI to complete legal tasks and correctly predicted the tasks for which GPT-4 were most helpful. These results have important descriptive and normative implications for the future of lawyering. Descriptively, they suggest that AI assistance can significantly improve productivity and satisfaction, and that they can be selectively employed by lawyers in areas where they are most useful. Because these tools have an equalizing effect on performance, they may also promote equality in a famously unequal profession. Normatively, our findings suggest that law schools, lawyers, judges, and clients should affirmatively embrace AI tools and plan for a future in which they will become widespread.
That is by Jonathan H. Choi, Amy Monahan, and Daniel Schwarcz, forthcoming in the Minnesota Law Review. Via the excellent Kevin Lewis.
