How Tessly cut alert noise across 400 endpoints

Fewer pages, faster response at Tessly

Too many alerts to trust

Tessly runs its platform in many cloud regions, and each region exposes its own APIs and endpoints. The previous monitoring setup checked them from one location. When that location had a network issue, dozens of healthy services paged at once. On-call engineers learned to wait a few minutes before acting, which slowed down response to real incidents.

Confirmation from three regions

With Navy, every endpoint is checked every 30 seconds from 18 regions. An outage is only confirmed when 3 regions see the same failure. In the first month, false alarm pages dropped by 92%, and engineers went back to treating every page as real.

Escalation that matches ownership

Each service team owns an escalation policy in Navy. Alerts go to the team’s Slack channel first, then to the primary on-call engineer through PagerDuty, then to the secondary after five minutes. Response time charts and the automatic incident timeline give everyone the same picture during an incident.

Details:

• Industry: Data & AI
• Location: San Francisco
• Employees: 5000+
• Identity Provider: Okta
• Key Workloads: Regional APIs, Workspaces, SQL Endpoints

Tessly now reviews Navy postmortems in its weekly reliability review. Real incidents are acknowledged in about two minutes, and new regions launch with monitoring in place from day one.