Anthropic Unveils Major Funding Program for Advanced AI Benchmarks

Anthropic aims to create an “early warning system” for identifying and assessing AI risks related to national security and defence.

Anthropic has launched a new initiative to fund the development of advanced benchmarks for evaluating AI models, including its own generative model, Claude. This program will provide funding to third-party organisations that can effectively measure advanced AI capabilities. Applications for funding are open on a rolling basis.

“Our aim with these evaluations is to enhance AI safety across the industry, providing valuable tools for the entire ecosystem. Creating high-quality, safety-relevant evaluations is challenging, and the demand for these evaluations is growing faster than the supply,” Anthropic stated on its blog.

AI currently faces significant benchmarking issues, with existing tests often failing to reflect real-world use cases. Many benchmarks, especially those developed before modern generative AI, may no longer accurately measure their intended metrics. Anthropic’s solution is to develop challenging benchmarks focusing on AI security and societal impacts through innovative tools, infrastructure, and methods.

Anthropic emphasises the need for tests assessing a model’s ability to perform tasks such as cyberattacks, enhancing weapons of mass destruction, and manipulating or deceiving people through deepfakes or misinformation. The company also aims to create an “early warning system” for identifying and assessing AI risks related to national security and defence, although specific details of this system were not disclosed.

Moreover, the program plans to support research into benchmarks and tasks that explore AI’s potential in scientific research, multilingual communication, bias mitigation, and self-censoring toxicity. Anthropic envisions new platforms allowing subject-matter experts to develop evaluations and large-scale model trials involving thousands of users. The company has hired a full-time coordinator for this program and may invest in or expand promising projects.