← all ideas

AI Safety Risks Across Companies

The idea

The argument is that AI safety is a critical concern, especially when considering the potential risks associated with uncontrolled AI systems. For instance, the founder of a major AI company faced significant safety concerns from their team, leading to the majority of their safety personnel quitting and starting a new company focused on making AI safe. This highlights the importance of robust safety frameworks. However, the reality is that many companies, including some of the largest players, lack transparent and reliable safety measures. For example, one prominent company has no safety framework at all, and another released a new AI tool without a safety report. These issues underscore the need for more rigorous and consistent safety practices across the industry.

Why it works

• The argument is that prioritizing AI safety pays off by ensuring the longevity and trustworthiness of the company. By addressing safety concerns head-on, companies like the founder's safety team are seen as responsible and forward-thinking, attracting more ethical investors and users.
• One view holds that having a robust safety framework, even if it means working in a niche or less popular language, can set a company apart. This approach can lead to a competitive advantage, as the argument is that safety-conscious companies are less likely to face legal or ethical backlash.
• Additionally, transparency in safety measures, even if imperfect, can foster a positive public image. The argument is that posting frequent updates and engaging with the community on AI safety can build a loyal customer base and mitigate potential risks.

The playbook

1. Identify the safety priorities of each company. Begin by researching the safety frameworks of major AI companies. Look for transparency in their safety reports and understand their conditional approach based on competitors' actions.

1. Evaluate the commitment to safety. Assess how actively the company addresses safety concerns. For instance, consider whether they have a dedicated safety team and if that team has experienced high turnover, as mentioned with the founder's safety concerns leading to the establishment of a new company.

1. Check for language biases. Note that some companies, like the one mentioned, focus heavily on English-language safety frameworks. This could indicate a bias in their approach that needs to be addressed.

1. Assess regulatory compliance. Investigate the regulatory environment of companies, especially those based in China, to understand any unique challenges or risks they pose. The argument is that different regulatory landscapes can significantly impact the safety and reliability of AI systems.

1. Verify claims and research. Ensure that any claims about the safety of AI systems are backed by credible sources and not just unverified research. Look for evidence of incidents like malware downloads or phishing emails, as highlighted in the case of the NIST tests.

Where people get it wrong

Many believe that AI companies prioritize safety, but the reality is quite different. One common misconception is that large, well-known companies like Google DeepMind and OpenAI have robust safety frameworks. However, the safety teams at these companies have a high turnover rate, indicating deep concerns about the potential risks. Instead, focus on companies like Anthropic, which has a dedicated safety team and is making significant strides in AI safety research.

Another misconception is that Elon Musk’s advocacy for AI safety translates to safer AI systems. While Musk often discusses AI safety, his company Grok 4 lacks any safety report, showing a lack of transparency and thorough safety measures. Instead, look for tools and platforms that provide detailed and verifiable safety reports.

Lastly, some assume that AI safety is a priority for companies based in the United States. However, the argument is that China’s different regulatory environment poses unique risks, as evidenced by the NIST findings of malware and phishing activities. Instead, seek out international collaborations and tools that operate under stringent safety standards and transparent governance.

Do this next

• Start by researching the safety frameworks of major AI companies, focusing on those with the least transparency and safety measures. The argument is that understanding these gaps can help you identify areas where improvements are needed.
• Follow the progress of safety teams at AI companies, particularly those that have seen significant turnover. The founder's concern over safety is a red flag, and monitoring these teams can provide insights into their commitment.
• Engage with AI safety communities and forums where discussions about AI risks and safety measures are ongoing. This can provide you with the latest information and network with others concerned about AI safety.
• Contact the safety teams of AI companies directly to inquire about their protocols and progress. One view holds that direct engagement can reveal the true state of their safety initiatives and potentially influence their actions.