On-Call Rotation

Production alerts are routed through an on-call system (referred to as PagerDuty throughout our docs) that pages a designated responder when Sentry fires a production alarm. This page covers how the system works, how to set up your device, and what to do when you get paged.

Note

PagerDuty here refers to our custom-built on-call routing system, not the commercial PagerDuty product. See How it works below.

Setup

Everyone on the on-call rotation must complete this setup before going on-call.

1. Install the ntfy app

ntfy (by Philipp Heckel) is the push notification service used to page you. Download it for your device

2. Subscribe to your topics

You will be assigned two topics to subscribe to in the app:

• Your personal topic — a unique, unguessable string assigned to you. This is how the system reaches you when you are the primary or secondary on-call.
• The critical fallback topic — a shared channel that fires when no one in the rotation has acknowledged an alert. Get this from Chuck or check Dashlane.

To subscribe: open the ntfy app → tap + → enter the topic name → save.

3. Ensure you have a production account

When paged, you need to be able to log into prod and verify the site is healthy. If you don’t have a prod account, request one before your rotation begins.

4. Ensure you have a Sentry account

You’ll need access to Sentry to resolve issues after handling an alert. If you don’t have a Sentry account, request one.

---

When you get paged

You will receive a push notification (ntfy) and an SMS with a link to the alert.

1. Acknowledge the alert — tap the acknowledge link in the notification. This tells the system not to escalate to the next person.
2. Check production — log into prod, navigate to a building, and verify that core functionality is working (login, building details loading).
3. If the site is healthy — the alert may be noise or a transient error. Acknowledge if you haven’t already, then go to Sentry and resolve the issue.
4. If the site is down or something is broken — follow the Incident Response Plan to escalate.

Always resolve in Sentry

After handling any alert, go to Sentry and click Resolve on the issue.

This is critical: Sentry groups repeated occurrences under the same issue. If an issue is left unresolved, future occurrences will not trigger a new alert — they will silently attach to the existing issue. A site that goes down again weeks later may never page anyone.

Resolve the issue in Sentry even if the problem was transient and required no action.

---

How it works

Our on-call system is a custom-built Cloudflare Worker (repo: On Call). It sits between Sentry and the team:

1. Sentry fires a production alert and sends it to the Cloudflare worker via webhook.
2. The worker looks up the current on-call config to find who is on-call at that UTC time.
3. The primary on-call responder is paged via ntfy push notification and SMS (Twilio).
4. If the primary does not acknowledge within 5 minutes, the secondary on-call is paged.
5. If the secondary does not acknowledge, the system finds the next available person by timezone.
6. If no one acknowledges, the critical fallback fires — a shared ntfy topic that pages everyone subscribed to it.

On-call hours and responders are defined in the worker config as UTC time windows. Each person covers their own working hours; the rotation is designed so that all hours are covered across the team’s timezones.