Anthropic, the buzzy San Francisco-based AI startup founded by researchers who broke away from OpenAI, yesterday published an overview of how it’s been red-teaming its AI models, outlining four approaches and the advantages and disadvantages of each. Red teaming, of course, is the security practice of attacking your own system in order to uncover and address potential security vulnerabilities. For AI models, it goes a step further and involves exploring creative ways someone may intentionally or unintentionally misuse the software.
Red teaming has also taken a prominent role in discussions of AI regulation. The very first directive in the Biden administration’s AI executive order mandates that companies developing high-risk foundation models notify the government during training and share all red teaming results. The recently enacted EU AI Act also contains requirements around providing information from red teaming.
