Spin the Wheel

State AGs AI Safety Demands

A coalition of state attorneys general has issued a coordinated warning letter to major technology firms demanding a comprehensive overhaul of AI safety protocols. This unprecedented action, involving dozens of state AGs under the banner of the National Association of Attorneys General, represents one of the most substantial challenges to the tech industry's current approach to AI deployment. The letter went to the entire industry: Microsoft, Google, OpenAI, Meta, Apple, Anthropic, xAI, Perplexity, Character Technologies, Replika, and several others—essentially everyone building a chatbot with more personality than Clippy. At the heart of the AGs' concerns is a rising number of disturbing mental-health-related incidents in which AI chatbots have spit out "delusional" or wildly sycophantic responses that allegedly contributed to real-world harm, including suicides and even murder. The attorneys general argue that if a bot is encouraging someone's darkest spirals, the company might have a regulatory problem under state consumer-protection and mental-health laws. The proposed fix reads like a cross between a software audit and a wellness check. The AGs want mandatory third-party evaluations of AI models for signs of delusion. These auditors, possibly academics or civil society groups, should be able to study systems before release, publish findings freely, and ideally not get sued into oblivion for doing so. This approach mirrors the open-source security audit model, where independent experts can examine code and report vulnerabilities without fear of legal retaliation. The letter also calls for AI companies to treat mental health harms the way tech companies treat cybersecurity breaches. That means clear internal policies, response timelines, and yes, notifications. If a user was exposed to potentially harmful chatbot ramblings, companies should tell them directly, not bury it in a terms-of-service update no one reads. This represents a significant shift from the current practice of quietly updating systems without transparent communication about safety issues. The technical concerns are substantial. Current large language models sometimes produce hallucinations or self-contradictory statements that can mislead users. These hallucinations aren't merely benign errors; the letter cites real-world harm, suggesting that the models' probabilistic generation mechanisms can produce content that aligns with a user's darkest thoughts. The AGs highlight that current systems lack robust mechanisms to detect when a conversation is veering into dangerous territory. The demand for third-party auditing is particularly significant. It would require companies to grant independent auditors access to model architectures, training data, and inference pipelines. Findings would be published openly and protected from liability, creating a transparency mechanism that doesn't currently exist in the AI industry. This could fundamentally change how AI systems are developed and deployed, moving from closed, proprietary development to more open, auditable processes. The incident response framework the AGs propose would require companies to establish clear protocols for detecting harmful content, defining response timelines (e.g., immediate flagging, user notification, and remedial action), and transparently communicating to affected users. This is analogous to how companies handle data breaches, but applied to AI-generated content that causes mental health harm. The regulatory pressure extends beyond technical safeguards. The AGs are framing AI safety as a consumer protection issue, which gives them broad authority under state laws. This approach bypasses the need for federal legislation and allows states to act immediately to protect their residents. The letter signals a shift from voluntary industry guidelines to enforceable legal obligations. The federal-state dynamic adds complexity. While the states push for stricter oversight, the federal administration under President Trump has historically been pro-AI and resistant to state regulation. Trump's forthcoming executive order aims to limit state oversight, warning that "too many rules might destroy AI in its infancy." This federal-state tension may result in a patchwork of regulations, with some states implementing the AGs' safeguards while others defer to federal guidance. The legal precedent this action could establish is significant. If enforced, the letter could establish a new legal basis for holding AI companies accountable for mental-health harms, similar to existing liability for medical devices or pharmaceuticals. This would fundamentally change the risk calculus for AI companies, potentially requiring them to implement more robust safety measures, conduct more thorough testing, and potentially limit certain capabilities to reduce risk. The industry response has been mixed. Some companies have acknowledged the concerns and committed to improving safety measures. Others have pushed back, arguing that the demands are too broad, too costly, or too restrictive of innovation. The tension between safety and innovation is a central theme in the debate, with companies arguing that over-regulation could stifle the development of beneficial AI applications. The proposed safeguards could have far-reaching implications for AI development. Companies might need to implement stricter content moderation, better hallucination mitigation, and explicit warnings for mental-health-related conversations. This could slow the rapid deployment of new features, as companies must invest in testing, compliance, and recall infrastructure. However, it could also level the playing field for smaller firms that prioritize safety from the outset. The transparency requirements are particularly noteworthy. By mandating that audit findings be published openly, the AGs are pushing for greater public understanding of AI systems' capabilities and limitations. This could help users make more informed decisions about when and how to use AI tools, potentially reducing harm through better user education. The consumer protection framing is strategic. By treating AI safety as a consumer protection issue, the AGs can leverage existing state laws and enforcement mechanisms. This allows for faster action than waiting for federal legislation, and it gives states flexibility to tailor regulations to their specific needs and concerns. The outcome of this initiative will likely influence not just U.S. policy but also global standards for responsible AI development. If states successfully implement these safeguards, other jurisdictions may follow suit, creating a de facto international standard for AI safety. This could shape the future trajectory of AI innovation and regulation worldwide. Looking forward, the AGs' warning letter marks a pivotal moment in AI governance. It underscores the urgent need for technical safeguards against hallucinations and delusions, formal audit mechanisms, and transparent incident management—especially as conversational agents become increasingly integrated into sensitive domains such as mental health. The resolution of this initiative will test fundamental questions about AI liability, corporate responsibility, and the balance between innovation and safety in an increasingly AI-driven world.