Nicolas Couvrat

Senior Software Engineer

Formal specifications can help engineers reason about complex distributed systems and verify critical correctness properties. But they remain valuable only as long as they reflect reality. At Datadog, we encountered this challenge while working on Husky, our event database. We had a formally verified model of Husky’s routing system that matched production behavior when we created it. Over time, however, Husky evolved, and a change to its placement logic allowed conflicting placements to persist in production for about a year. Although harmless during normal operation, just two conflicting placements would have caused Husky’s backfill system to crash while recovering from placement data loss. We found the issue not by triggering that failure, but by checking production behavior against our TLA+ specifications.