Skip to content
Cadence OneFive° Internal Handbook

SLOs and Monitoring Strategy

Current State (As of October 2023)

Section titled “Current State (As of October 2023)”

Where we are:

  • No formal SLOs except when stakeholders escalate issues
  • No structured incident management process (ad hoc Slack monitoring/alerting)
  • Uptime Robot is our only automated monitoring

Gaps:

  • No well-defined Service Level Indicators (SLIs) supporting SLOs
  • No automated alerting system
  • No public incident tracking
  • No post-mortem process

Vision: Where We Want to Be

Section titled “Vision: Where We Want to Be”

Well-Defined SLIs and SLOs

Section titled “Well-Defined SLIs and SLOs”

Each Service Level Objective (SLO) should be supported by measurable Service Level Indicators (SLIs):

  • Availability SLI: Uptime percentage (e.g., 99.9% uptime)
  • Latency SLI: Response time percentiles (e.g., p95 < 500ms)
  • Error Rate SLI: Percentage of failed requests (e.g., < 0.1% error rate)
  • Throughput SLI: Requests per second capacity

Automated Monitoring and Alerting

Section titled “Automated Monitoring and Alerting”

Each SLI should have:

  • Automated monitors that trigger alerts when thresholds are breached
  • Alert routing to SquadCast/PagerDuty for on-call engineers
  • Public incident creation in company Slack for visibility and investigation

Incident Management Process

Section titled “Incident Management Process”

For every incident:

  1. Track publicly - Create incident thread in company Slack
  2. Respond systematically - Follow documented response procedures
  3. Learn from mistakes - Write post-mortem for every incident
  4. Prevent recurrence - Goal is to make each mistake only once

Incident Response Framework

Section titled “Incident Response Framework”

Severity Levels

Section titled “Severity Levels”

Define incident severity based on impact:

  • SEV-1 (Critical): Complete service outage, revenue impact
  • SEV-2 (High): Partial outage, degraded performance
  • SEV-3 (Medium): Minor issues, workarounds available
  • SEV-4 (Low): Cosmetic issues, no user impact

Response Process

Section titled “Response Process”
  1. Detect - Automated monitoring or user report
  2. Alert - Notify on-call engineer
  3. Investigate - Gather data, identify root cause
  4. Resolve - Implement fix or workaround
  5. Communicate - Update stakeholders
  6. Document - Write post-mortem

Post-Mortem Template

Section titled “Post-Mortem Template”

Every incident should have a post-mortem including:

  • Timeline - When did it start? When was it detected? When was it resolved?
  • Impact - How many users affected? Revenue impact? Duration?
  • Root Cause - What actually broke?
  • Resolution - How was it fixed?
  • Action Items - What will prevent this from happening again?
  • Owner - Who is responsible for each action item?

Monitoring Tools

Section titled “Monitoring Tools”

Current Tools

Section titled “Current Tools”
  • Uptime Robot - Basic uptime monitoring
  • Sentry - Error tracking and debugging
  • Fly.io Metrics - Infrastructure monitoring

Future Considerations

Section titled “Future Considerations”
  • SquadCast/PagerDuty - On-call scheduling and alerting
  • Datadog/New Relic - APM and infrastructure monitoring
  • Grafana/Prometheus - Custom dashboards and metrics

Getting Started

Section titled “Getting Started”

Define Your First SLO

Section titled “Define Your First SLO”

Start with one critical service:

  1. Identify the service - What’s most critical to users?
  2. Choose an SLI - What metric matters most? (uptime, latency, errors)
  3. Set a target - What’s acceptable? (99.9% uptime, p95 < 500ms)
  4. Implement monitoring - Add automated checks
  5. Configure alerts - When should someone be notified?

Iterate and Improve

Section titled “Iterate and Improve”
  • Start small, expand coverage over time
  • Review SLOs quarterly - are they too strict or too lenient?
  • Adjust based on actual incident data
  • Build a culture of reliability through consistent post-mortems
Section titled “Related Resources”

External References

Section titled “External References”