A controversial bill that would require developers of advanced AI models to adopt safeguards is one step closer to becoming law.

The bill (SB 1047) would require developers of future advanced AI models to create guardrails to prevent the technology from being used to launch cyberattacks on critical infrastructure like power plants.

Developers would be required to submit safety plans to the Attorney General, who could hold them liable if AI models over which they have direct control pose harm or imminent threats to public safety.

The bill, introduced by Sen. Scott Wiener (D-San Francisco), passed the state Assembly on Wednesday with 41 votes in favor and 9 against. It was passed overwhelmingly by the state Senate on Thursday. The bill now heads to Gov. Gavin Newsom's office, but it's unclear whether he will sign the bill or veto it.

“Innovation and safety can go hand in hand, and California is leading the way,” Wiener said in a statement.

A spokesman for Governor Newsom said the bill will be evaluated once it reaches his desk.

Wiener's bill has sparked fierce debate in the Bay Area tech community, with support from the Center for AI Safety, Tesla CEO Elon Musk, the Los Angeles Times editorial board and San Francisco-based AI startup Anthropic, among others.

However, Democratic congressional leaders and prominent AI companies such as Meta and OpenAI opposed the bill, citing concerns that it would stifle innovation in California.

Democratic congressional leaders, including former House Speaker Nancy Pelosi, Rep. Ro Khanna (D-Fremont), and Rep. Zoe Lofgren (D-San Jose), also opposed the bill and urged Governor Newsom to veto it. They argued that the bill could harm California's growing AI industry, home to ChatGPT developer OpenAI, and cited Congressional efforts related to AI.

“There is a real risk that companies will incorporate in other jurisdictions or decide not to release their models in California,” Khanna, Lofgren and six other Democratic Assembly members said in a letter to Governor Newsom.

“We want California to lead the way on AI in a way that protects consumers, data, intellectual property and more, but SB 1047 does more harm than good in that pursuit,” Pelosi said in a statement.

Wiener and other supporters of the bill disagree, arguing that the bill would protect people while encouraging innovation.

“There have to be guardrails in place,” Rep. Devon Mattis (R-Visalia) said before the House vote Wednesday afternoon. “We have to make sure they're a responsible player.”

Supporters of SB 1047 argue that the bill would hold developers responsible for the safety of the advanced AI models they control, potentially helping to prevent catastrophic AI events in the future.

“I worry that technology companies are so caught up in the race to maximize market share and profits that they will not be able to address the significant risks associated with AI on their own,” Yoshua Bengio, a professor at the University of Montreal and founder and scientific director of the Quebec Institute for Artificial Intelligence Mira, said at a press conference this week. “We can't just let them grade their own homework and expect it to work.”

Advocates also argue that AI should be regulated like other industries that pose potential safety risks.

“This is a difficult call and it will anger some people, but all things considered, I think California should probably pass SB 1047, the AI ​​Safety Bill,” Musk wrote on X on Monday. “For over 20 years, I have advocated for regulating AI, just as I would any product or technology that poses a potential risk to the public.”

Earlier this month, the bill passed a key state Senate committee after Wiener made significant changes, including removing perjury penalties and changing the legal standard for developers regarding the safety of advanced AI models.

Dario Amodei, CEO of San Francisco-based AI startup Anthropik, said in an Aug. 21 letter to Newsom that he believes the bill's “benefits likely outweigh the costs. The letter does not endorse the bill but shares the company's perspective on the pros and cons.

“I want to be clear that SB 1047 addresses real and serious concerns about the catastrophic risks of AI systems,” Amodei wrote. “The capabilities of AI systems are advancing extremely rapidly, offering both great promise and great risks to California's economy.”

But some technology companies, like OpenAI, said they still opposed the bill even after the changes.

“Given the broad and significant impact that AI will have on U.S. competitiveness and national security, regulations for the Frontier Model must be developed and implemented at the federal level,” Jason Kwon, chief strategy officer at OpenAI, wrote to Wiener on Aug. 21. “A federally led set of AI policies, rather than a patchwork of state laws, would foster innovation and position the United States to lead the development of global standards.”

Wiener said he would welcome strong federal AI safety legislation that would supersede his bill.

“Past experience suggests that enacting such legislation will be an uphill battle,” Wiener said in a statement. “In the meantime, California should continue to lead the way with policies like SB 1047 that protect our citizens while encouraging innovation.”

SB 1047 is one of roughly 50 AI-related bills in the state Legislature dealing with various aspects of how the technology affects the public, including jobs, deepfakes and safety.