A monitoring product has to be more reliable than the services it watches. The best way we know to get there is to use Navy for everything we run. Here is how we monitor Navy with Navy, and what it has taught us.
1. Every service has a monitor in code
Our check runners, alert delivery, API and status pages are all defined in Terraform with the Navy provider. A new service cannot go live without its monitors in the same pull request. We have not created a production monitor by hand in over a year.
2. A separate stack watches the main one
Monitoring yourself has an obvious flaw: if Navy is down, it cannot tell you. We run a small, independent Navy stack on different infrastructure whose only job is to watch the main one. Each stack checks the other.
3. Heartbeats on every background job
Billing, data retention, certificate checks and subscriber emails all run as scheduled jobs. Each one sends a heartbeat when it starts and finishes. A missed heartbeat pages the same rotation as an API outage.
4. Our status page is a Navy status page
Our public status page runs on the same product as yours, with the same subscriber emails and 90-day uptime bars. If something in the status page flow is confusing during an incident, we feel it first and fix it.
5. What it has taught us
Using our own alerts every day has shaped the product more than any roadmap. Escalation policies, on-call schedules and the automatic incident timeline all started as fixes for our own on-call.
| System | Monitor types | Alert route |
|---|---|---|
| Check runners | Heartbeat, TCP | On-call |
| Alert delivery | HTTP, keyword | On-call |
| Status pages | HTTP, SSL certificate | On-call and Slack |
Running Navy on Navy keeps us honest. Every alert we send is one we would want to receive, and every rough edge in the product is one we hit ourselves first.
