In our first post, How to evaluate AI SRE agents for production, we evaluated OpenSRE and scored it on five dimensions:

  • Actionability and control safety
  • RCA quality
  • Integrations and correlation
  • Incident memory
  • Security

Across these dimensions, we cared about two questions:

  • Can the agent provide the on-call engineer with a reliable answer based on the available evidence during an incident?
  • If yes, can that answer reduce the time to mitigate the issue safely?

Some of the dimensions in the OpenSRE PoC were too broad to show what went wrong in an investigation. In one test, we intentionally scaled replicas to zero using kubectl, and OpenSRE blamed a Helm rollback as the root cause with no evidence behind it. We scored this RCA -1.