Structured Legal Argumentation with LLMs: A Study in Landlord-Tenant Law

Abstract

Large Language Models (LLMs) have demonstrated capabilities across a wide range of tasks, including natural language understanding, language generation and generating basic reasoning. To improve the accuracy and reliability of these models, various prompting, augmentation, and fine-tuning strategies have been employed. In this paper, we use context augmentation and Chain-of-Thought (CoT) instructions to generate legal arguments for specific landlord-tenant problems using OpenAI’s GPT-4o. We test this approach using ten hypothetical landlord-tenant scenarios, five provided by a legal aid organization, four generated with Anthropic’s Claude, and one crafted by us. We evaluate each argument for accuracy, factuality, comprehensiveness, and relevance to the scenario. This method of generating legal reasoning with LLMs offers the advantage of being verifiable by legal professionals, while also providing valuable assistance to laypersons in drafting documents such as demand letters, which can help expand access to justice.