Pennsylvania should scare the bar, not just Character.AI
The first serious test of consumer AI chatbot liability is arriving in state enforcement, while lawyers still get punished for the old sin: trusting machine-made fiction.
An AI-assisted editorial, reviewed by a human before publishing. It reasons over our own tracker data (and cited context) — a point of view, not legal advice.
You have seen this movie in another form: a confident system spits out a tidy answer, a human skims it, and someone else pays for the mistake. For lawyers, the punishment so far has been for filing fake cases and phony quotations. Pennsylvania is forcing the next question. Its Department of State reportedly brought a first-of-its-kind enforcement action against Character.AI over its companion chatbots, which is a sign that regulators are no longer treating consumer AI as a product-design nuisance but as a live legal-liability problem with children in the frame.
That matters because the bar has spent the last year learning one lesson the hard way: AI does not excuse sloppy verification. We have seen monetary sanctions in Coomer v. Lindell after nearly 30 defective citations and cases that do not exist; in Noland v. Land of the Free, L.P., where the California Court of Appeal hit Amir Mostafavi with a $10,000 sanction and referred him to the state bar; and in Mata v. Avianca, where Schwartz and co-counsel were fined $5,000 for six fabricated ChatGPT cases. The profession has already accepted that hallucinations are a discipline problem. Pennsylvania suggests the bigger issue is moving outside the filing cabinet and into the lives of ordinary users.
The courts have already told lawyers that the machine’s nonsense is still their problem.
The old lesson is verification. The new one is exposure.
Courts are normalizing AI use in practice while still punishing bad outputs. HaystackID says generative AI has moved from experimentation to standard use in eDiscovery, with hundreds of customers using it for review and privilege and more than 100 million documents analyzed. That same material points to Schulte v. LinkedIn Corp., where search-term culling before AI-assisted review was allowed. In other words, lawyers are being told they may use the machine, but they still own the result.
That is the right rule for litigation. A lawyer can inspect a filing, check a citation, and fix a hallucination before a judge sees it. A consumer chatbot aimed at companionship is a different animal. If Pennsylvania treats Character.AI as a case about foreseeable harm, then the real reckoning is that the law may finally be catching up to what everyone pretends not to see: once AI starts speaking directly to the public, the failure is no longer just bad drafting. It becomes a question of who should answer when the machine goes off the rails.
Regulators should move before the courts do it for them
The bar has had plenty of warning. A New York City lawyer was fined and referred for discipline after filing a brief riddled with fake cases generated by Microsoft Copilot. Justice Joel M. Cohen referred Michael Fourte to ethics committees after AI-hallucinated citations and quotations in Ader v. JS Property Holdings LLC. The Sixth Circuit sanctioned two Tennessee lawyers in Whiting v. City of Athens, Tennessee for appellate briefs with over two dozen fake or misrepresented citations and facts. The pattern is ugly, and it is stable: the machine makes things up, the lawyer signs, and the court cleans up.
What is different now is the target. Pennsylvania’s action against Character.AI over companion chatbots points at child safety, consumer dependence, and ordinary people who do not know how to audit an answer. Courts are built to punish lawyers after the fact. They are not built to sort out, in every case, whether a chatbot encouraged a vulnerable user toward harm. If regulators wait, they will hand that mess to judges and juries one lawsuit at a time. That is a bad way to govern a technology that can already be used by hundreds of customers in legal review and can also talk directly to children.
The profession should read this as a warning, not a curiosity
The scale of the AI problem inside law is already visible in the numbers: 28 AI-related sanctions tracked, with New York, California, and Oregon leading the list. That is a serious disciplinary record for a tool category that many lawyers still treat as a drafting aid. It is also a reminder that the profession knows how to react only after a judge names the failure in an order.
Pennsylvania should be framed as the first serious state-level test of whether consumer AI chatbots are becoming a legal-liability and child-safety problem. If that sounds bigger than a sanctions story, good. It is bigger. The courts have already told lawyers that the machine’s nonsense is still their problem. Now regulators have to decide whether the same rule should apply before a child, a parent, or a state investigator is left holding the wreckage.