Incident Response Plan

This document offers guidance and common vocabulary for responding to incidents, whether related to security, or the behavior of the product.

TL;DR: If something bad happens, create a post in #tech_incidents (forum channel) with the appropriate severity tag, communicating your assessment of severity (Really Bad stuff is Critical Severity - Sev 1, or High Severity - Sev 2). Escalate based on severity. The response team should follow the protocol below.

For the step before this — determining whether an alarm warrants an incident — see the Sentry Alarm Triage guide.

Channels

For alarm channels and environment coverage, see Sentry Alarm Triage.

#tech_incidents is a forum channel where incidents are manually created with appropriate tags (sev-1, sev-2, active, resolved, security, outage). Each incident becomes a threaded discussion.

First Responder Role

The First Responder rotates by sprint.

Responsibilities:

• Monitor #alarms_production for automated alerts — PagerDuty pages the on-call First Responder during their defined on-call hours, escalating to secondary and beyond outside those hours (see On-Call Rotation)
• First in line to triage issues - when an alert warrants an incident, create a post in #tech_incidents with the appropriate severity tag (link to or quote the original alert)
• Join bi-weekly Bugs & Sentry Backlog meeting and report on Sentry
• Responsible for deployments

Escalation

Create a new post in #tech_incidents to notify the team of issues. Be a good witness. Behave as if you were reporting a crime and include lots of specific details (links, time of observation, repro steps) about what you have discovered, as well as your assessment of the severity of the issue.

Severity

Assigning severity is not an exact science, and reasonable folks can disagree. Having a common vocabulary is a way to reduce confusion, gradually align on what constitutes an emergency, and accelerate responses. Judgment always trumps definitions.

Sev-1: Critical Severity

Critical issues relate to actively exploited security risks, or site issues with a massively negative impact on the business. This should be extremely rare, and resolving the issue would be the highest priority for the entire company.

Example: a malicious actor is actively leveraging a security vulnerability, confidential user information is compromised.

Reporting/Escalation: Critical severity issues should involve creating a post in #tech_incidents with the sev-1 tag, and pinging @everyone in the post. Also send direct messages to Bomee and Marc for awareness. Continue escalation (chat or phone) until you receive acknowledgement that the team is on it.

Response Expectation: stakeholders should give Sev-1s absolute priority over any other task, and justify e.g. phone calls at 2AM.

Sev-2: High Severity

High severity issues relate to production site problems that severely impact our users ability to use our systems, or security vulnerability where an adversary or active exploitation hasn’t been proven yet, and may not have happened, but is likely to happen.

Examples:

• a critical functionality of the site is not accessible [during NY business hours], making it unusable.

• security vulnerabilities are discovered (eg: backdoors, malware, malicious access of business data)

Reporting/Escalation: High severity issues should involve creating a post in #tech_incidents with the sev-2 tag, and pinging @everyone in the post. Continue escalation until you receive acknowledgement that the dev team is on it.

Response Expectation: the response owner(s) should give Sev-2s absolute priority over any other of their tasks, and ask for help to expedite resolution.

Sev-3 and Sev-4: Medium and Low Severity

Issues meeting this severity are simply suspicions or odd behaviors. They are not verified and require further investigation. There is no clear indicator that systems have tangible risk. This includes suspicious emails, outages, strange activity on a laptop, site issues that users can easily work around.

Examples:

• Sev-3: Links to advanced search are gone, but users can access them via direct urls.

• Sev-4: The release documentation page isn't accessible on the site.

Reporting/escalation: these issues can be processed via standard bug triage, or messages to the relevant chat channels (not #tech_incidents).

Response Expectation: these issues do not require emergency response.

Response Steps

For both sev-1 and sev-2 issues, the response team will follow an iterative response process designed to investigate, contain exploitation, remediate our vulnerability, and document a post-mortem with the lessons of the incident.

1. The person who is initially responding to the incident will take on the role of Incident Lead. They may choose to pass this baton to another person as the incident proceeds (e.g. if somebody with more experience becomes available.)
2. When a member of the dev team has assessed that the issue warrants sev-1 or sev-2 severity, the Incident Lead should create a post in #tech_incidents with a descriptive title (e.g. “War Room: 500 errors on login”), appropriate severity tag, and the active tag. The post becomes a thread where stakeholders can coordinate and track progress.
3. The team should first focus on collecting and sharing information. The Incident Lead may direct participating team members to look at different information sources.
4. The Incident Lead should, throughout the incident, determine whether the right team is gathered to work on the problem. If specific individuals are needed, they can attempt to contact them to ask them to join. Do not hesitate to pick up the phone and call somebody if the incident is of sufficient urgency.
5. As theories develop, the Incident Lead should try to prioritize, and divide the available team members to work on different investigatory threads. They may check in with each group periodically via voice call, or by messaging in the thread. They may also simply request that each group report back every 5 or 10 minutes with a quick status update.

Making Changes

1. It’s important that changes are made in a controlled manner. Before any change is applied, it must follow our regular change control process. It’s very easy to make a bad situation worse through a series of panicked changes which aren’t documented. Breathe!
2. Whenever possible, changes should be applied to and evaluated in the staging environment.

On Incident Resolution

The Incident Lead should:

1. If the root cause has still to be identified, create a ticket for followup, assign it, and mark it with the appropriate priority (usually “high”).
2. Post a final message in the incident thread summarizing the resolution and whether the root cause has been identified. Update the post’s tags: remove active and add resolved.
3. Update the Incident Log with a short summary of what occurred and what the solution was, so that if a similar issue occurs in future, the resolution is faster.
4. Add an agenda item for the upcoming architecture meeting, in order to brief the team on what happened.

The team member “closest to the issue” should:

1. Instantiate a post-mortem doc soon after the incident is resolved.

Response Team Members

All software developers for all products are part of the tech incidents team. Phone numbers for all staff are securely available to all on Justworks. If chat is down, call Bomee and François

Incident Log

We maintain post-mortems in the incident-post-mortems GitHub repo.

Related Documents

• Corrective Action Procedure — When a security incident reveals a nonconformity in the ISMS, the corrective action process is used to evaluate root cause and prevent recurrence