A federal judge sanctioned an attorney for citing six cases that did not exist. Another firm was fined after submitting a brief built on fabricated precedent generated by a chatbot. These are not isolated incidents anymore. They are a pattern, and the pattern keeps repeating because lawyers are adopting AI research tools faster than they are adopting the habits needed to use them safely.
This is not an argument against AI in legal research. The technology is genuinely useful, and the tools reshaping how lawyers find and analyze legal information are only getting better. But useful and safe are not the same thing, and the gap between them is where careers get damaged.
Here is what you need to understand before you let AI anywhere near a brief, a memo, or a client recommendation.
AI Hallucinations Are Not Rare Glitches. They Are a Built-In Feature.
Large language models do not retrieve information the way a database does. They generate text by predicting what word is statistically likely to come next, based on patterns learned during training. Most of the time, this produces accurate, useful output. Sometimes, with total confidence and perfect formatting, it produces something completely false.
A hallucinated case citation looks exactly like a real one. Correct case name format, plausible court, a citation number that follows the right pattern. The only way to know it does not exist is to check it. This is the trap. The fabricated output is indistinguishable from real output by appearance alone, which means every single citation an AI tool produces needs independent verification before it goes anywhere near a filing.
Why Verifying Every AI-Generated Citation Is Now a Bar Requirement
Several state bars have issued formal guidance making clear that an attorney’s duty of competence extends to understanding the tools they use, including AI. That means you cannot blame the chatbot if a hallucinated case ends up in your filing. Courts have made the same point directly, sanctioning attorneys regardless of whether they personally wrote the fabricated content or simply failed to catch it. Verification is not optional, and it is not the paralegal’s problem to quietly solve. It is yours.
Not All AI Tools Are Built for Legal Research, and the Difference Matters
There is a meaningful gap between a general-purpose chatbot and a legal research platform with AI built in. Tools like Westlaw’s AI-powered features, Lexis+ AI, and Casetext’s CoCounsel are trained on and connected to actual legal databases. When they cite a case, they are pulling from a verified source, not generating plausible-sounding text from scratch. That does not make them infallible, but it puts them in a fundamentally different risk category than asking a general AI model to summarize case law from memory.
General-purpose tools like ChatGPT, Gemini, or Claude can be genuinely useful for legal work: summarizing a document you provide, brainstorming arguments, explaining a concept in plain language. What they should not be used for is generating case citations from their own training data without verification against a real legal database. The training data has a cutoff date, gaps, and no live connection to current case law. Treat output from these tools as a draft from a smart but unreliable research assistant, not as a finished citation you can drop into a filing.
How to Choose the Right AI Tool for the Legal Research You Are Doing
Match the tool to the task. Use legal-specific platforms connected to verified databases for anything involving case citations or statutory research. Use general AI tools for drafting support, summarization, and brainstorming, with the understanding that anything resembling a legal fact needs to be checked against a primary source before it leaves your desk.
Confidentiality Does Not Pause Because You Are Talking to a Chatbot
Every detail you type into an AI tool, including case facts, client names, and strategy notes, may be processed, stored, or in some cases used to train future versions of the model, depending on the platform and your settings. For an industry built on attorney-client privilege, this is not a minor technical detail. It is a direct ethical exposure.
Several bar associations have issued guidance specifically addressing this. The core message is consistent: confirm exactly how a given AI tool handles the data you input before you input anything sensitive. Enterprise-grade legal AI platforms typically offer data protection commitments that consumer-facing chatbots do not. The free version of a general AI tool and the enterprise version of the same company’s product can have completely different data handling policies, and the difference is not always obvious from the interface.
What to Check Before Typing Client Information Into Any AI Platform
Read the data retention and training policy for any AI tool before using it with real client information. Ask whether the platform offers a business or enterprise tier with stronger confidentiality commitments. When in doubt, anonymize the facts before you input them, or do not input them at all. This is the same caution you would apply to any third-party vendor handling privileged information, and AI tools deserve no exception.
The Lawyers Getting This Right Treat AI as a First Draft, Never a Final Answer
The firms avoiding sanctions and embarrassing corrections share a common workflow. AI generates a starting point. A human verifies every factual claim and every citation against a primary source. Nothing goes into a filing or a client communication without that second step.
This same discipline applies well beyond the courtroom. AI tools are increasingly summarizing and citing information about law firms themselves when potential clients search for legal help, and the same hallucination risk that can put a fake case into your brief can put inaccurate information about your firm in front of a potential client. The accuracy problem with AI does not stay contained to one use case. It shows up everywhere the technology touches your practice.
If your firm is thinking seriously about where AI fits into legal work, it is worth thinking just as seriously about where AI fits into how clients find and evaluate your firm in the first place. Both problems come from the same root cause: AI systems that sound confident regardless of whether they are right.
Bring AI Into Your Practice Carefully, Not Reluctantly
None of this is a reason to avoid AI research tools. Firms using them well are moving faster, covering more ground, and catching connections that manual keyword searches miss. The lawyers running into trouble are not the ones using AI. They are the ones using it without a verification habit, without understanding what data they are exposing, and without a clear sense of which tool is appropriate for which task.
Civille works with law firms navigating exactly this shift, not just in legal research but in how AI is reshaping the entire client journey, from the first search query to the final retained client. Our digital marketing team helps firms understand where AI creates opportunity and where it creates exposure, both inside the practice and in how the firm shows up online.
If you want a clearer picture of how AI is affecting your firm beyond the research desk, talk to the Civille team and request a demo.
