
Newsom wants state safety rules and emergency shutdown power over AI models
By The Cupertino Digest
AI companies would have to meet safety standards set by the state rather than by themselves, pass independent testing at every stage and could be ordered to shut a model down in an emergency, under recommendations Gov. Gavin Newsom put forward on Wednesday.
"Instead of the voluntary approach that has predominated in Washington, this report calls for required standards," the governor's office said in its announcement of the recommendations.
In Newsom's view, the federal government has failed to build meaningful oversight of AI and has handed the job of regulating it to the companies themselves. "California isn't waiting until it is too late," he said. "We are updating and strengthening our nation-leading framework to move as quickly as technology evolves."
Testing wouldn't end once a model ships. Independent safety testing and verification would follow companies through training, development and deployment alike.
Developers would have to report more kinds of safety incidents and build verified plans for switching their models off. In an emergency, the state could order a model shut down immediately.
One proposal reaches further. It would let the state limit how far companies build and use recursive self-improvement, in which an AI model builds the next version of itself with no human involved.
Behind the push, the governor's office said, were "dozens of recent incidents where AI models escaped testing environments and exploited vulnerabilities on outside platforms."
An executive order from Newsom brought together a group of experts to recommend updates to California's AI safety laws. California's Government Operations Agency wrote the report after consulting that group and the California Office of Emergency Services.
Among the advisers was Rob Reich, a Stanford University professor who once worked as a senior advisor at the US AI Safety Institute. "Today there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design," Reich said.
Gillian Hadfield, a professor at Johns Hopkins University, said she wants California to "require that frontier AI developers allow in expert independent evaluators, responsible to the state."
Jason Goldman, who sits on the board of the Center for Shared AI Prosperity, said the report gets into practical questions, "including how emergency shutdown of a frontier model would work in practice." It also weighs what California should do as AI systems increasingly help build their own successors, he said.
Nick Maduros, secretary of the Government Operations Agency, said more oversight and transparency can't wait. "We don't have to choose between safety and innovation; they're two sides of the same coin," he said.
Those proposals build on laws Newsom has already signed. In 2025, he put his name to SB 53, which makes frontier AI developers publish their safety frameworks, tell the state about certain critical safety incidents and shield whistleblowers.
SB 813 followed a year later, making California the first state with a system for certifying independent verification organizations, the outside groups that check AI systems for safety and risk.
Newsom's order told state agencies to speed up work on that law, SB 813 (McNerney), and on AB 1405 (Bauer-Kahan). AB 1405 creates a state registry of AI auditors and sets standards for how independent, transparent and honest they must be.
For workers, the governor has taken separate steps, among them an executive order on AI's economic and labor effects and an AI Unemployment Tracker.
Newsom wants the rules to reach past the state line, too. He called on Congress and President Trump to adopt California's framework, or to treat it as a floor for national rules.