开源基准 CloudOpsBench 显示推理是 AI 云运维的最大瓶颈
New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
AI 导读
密歇根大学 Ang Chen 团队发布开源基准 CloudOpsBench 1.0，用 45 个 AWS 运维任务评测 AI 智能体，涵盖修复服务、删除资源、限制权限、推送更新和下线项目。
摘要与补充说明
表现最好的 Claude Fable 5.1 完成 60%，GPT-6 Astra 完成 53%，没有模型完成全部任务。团队称主要瓶颈不是知识不足而是推理，约 80% 的失败源于智能体误判任务范围、未检查所有相关服务或过早停止，另有约 20% 是错误操作，部分对基础设施造成不可逆后果。
正文
CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, from fixing services and removing resources to restricting access rights, rolling out updates, and decommissioning projects. Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers.
Even the best models fail at many of the tasks. Claude Fable 5.1 solved 60 percent, GPT-6 Astra managed 53 percent. According to the team led by Ang Chen at the University of Michigan, the main problem isn't a lack of knowledge but reasoning. About 80 percent of failures happen because agents misjudge the scope of a task, don't check all the relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for the infrastructure.
All tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned. The project on GitHub is licensed under Apache 2.0, and a detailed failure analysis is also available.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.