2024-2025 Incident Report Summary
This document summarizes all production incidents from Slack war room channels.
Total Incidents: 9
Summary Table
Section titled “Summary Table”| Date | Incident Type | Duration | Messages | Resolved | Notes |
|---|---|---|---|---|---|
| 2025-06-12 | incident | 13m | 7 | ✅ | |
| 2025-05-06 | site down | 3h 11m | 83 | ✅ | Has follow-up discussions |
| 2024-10-25 | gateway timeouts in prod | 7007h 2m | 37 | ❓ | Has follow-up discussions |
| 2024-10-08 | simulation incident | 20m | 12 | ✅ | |
| 2024-08-21 | prod down | 29m | 4 | ✅ | |
| 2024-06-27 | production 500s | 1h 36m | 28 | ✅ | Has follow-up discussions |
| 2024-05-23 | ssl handshake failed | 34m | 5 | ❓ | |
| 2024-03-14 | blank screen home | 26m | 14 | ✅ | |
| 2024-01-17 | site incident edit details | 46h 18m | 22 | ✅ | Has follow-up discussions |
2025 Incidents Timeline
Section titled “2025 Incidents Timeline”Visual timeline showing incident duration (scaled for visibility):
05-06 site down ████████████████████████████████████████████████████████████ 3h 11m06-12 incident ████ 13m2025 Incidents Summary
Section titled “2025 Incidents Summary”| Date | Type | Duration | Status |
|---|---|---|---|
| 2025-05-06 | site down | 3h 11m | ✅ Resolved |
| 2025-06-12 | incident | 13m | ✅ Resolved |
2025 Incidents
Section titled “2025 Incidents”2025-06-12 - incident
Section titled “2025-06-12 - incident”- Duration: 13m
- Messages: 7
- Resolved: Yes
- Channel:
2025-06-12-incident - Summary: We’re in this meet https://meet.google.com/ccr-ksye-nxc <@U067U8GMY02> some fun for when you wake up I’ve seen two emails so far about folks not being able to get into MOmentum.
One user from the W…
2025-05-06 - site down
Section titled “2025-05-06 - site down”- Duration: 3h 11m
- Messages: 83
- Resolved: Yes
- Channel:
2025-05-06-site-down - Summary: hi http://meet.google.com/myp-ogjy-pcb|meet.google.com/myp-ogjy-pcb background: https://cadence15.slack.com/archives/C05TNCQLJ86/p1746549065335199 <mailto:cadence-610@support.fly.io|cadence-610@su…
2024 Incidents
Section titled “2024 Incidents”2024-10-25 - gateway timeouts in prod
Section titled “2024-10-25 - gateway timeouts in prod”- Duration: 7007h 2m (tracking channel, not a single incident)
- Messages: 37
- Resolved: Unknown
- Channel:
2024-10-25-gateway-timeouts-in-prod - Summary: channel to track the issue where we’re getting gateway time outs in prod The first gateway timeout discovered from the click around last night was the “the bronx” problem- (we deemed non-block…
- Note: This appears to be a tracking channel for ongoing issues rather than a single incident. Duration reflects time between first and last message in the channel.
2024-10-08 - simulation incident
Section titled “2024-10-08 - simulation incident”- Duration: 20m
- Messages: 12
- Resolved: Yes
- Channel:
2024-10-08-simulation-incident - Summary: Report from Jason: We’re receiving multiple customer complaints about slow API responses and some users are reporting seeing other users’ data intermittently. Is anyone else seeing this? Nothing usefu…
2024-08-21 - prod down
Section titled “2024-08-21 - prod down”- Duration: 29m
- Messages: 4
- Resolved: Yes
- Channel:
2024-08-21-prod-down - Summary: http://meet.google.com/ipp-kthi-hpd|meet.google.com/ipp-kthi-hpd Re-running yesterday’s release deploy steps to bounce the machines. GH action <https://github.com/CadenceOneFive/momentum/actions/r…
2024-06-27 - production 500s
Section titled “2024-06-27 - production 500s”- Duration: 1h 36m
- Messages: 28
- Resolved: Yes
- Channel:
2024-06-27-war-room-production-500s - Summary: first! Is this the right thread for background? https://cadence15.slack.com/archives/C05TNCQLJ86/p1719447684004379 Here is a video with symptoms I was seeing this morning: <https://cadence15.slack.c…
2024-05-23 - ssl handshake failed
Section titled “2024-05-23 - ssl handshake failed”- Duration: 34m
- Messages: 5
- Resolved: Unknown
- Channel:
2024-05-23-war-room-ssl-handshake-failed - Summary: I’m taking a look at the fly logs following deploy. https://fly.io/apps/momentum-production/monitoring ```> fly certs show http://momentum.c15.io|momentum.c15.io -a momentum-production The cert…
2024-03-14 - blank screen home
Section titled “2024-03-14 - blank screen home”- Duration: 26m
- Messages: 14
- Resolved: Yes
- Channel:
2024-03-14-war-room-blank-screen-home - Summary: <@U059LHGBHM4> - does it make sense to start a voice only huddle? <https://github.com/CadenceOneFive/momentum/commit/2e1c5abb0b74b090de16e81b525db7e17f43707d|Likely related PR>. <@U059LHGBHM4> do you …
2024-01-17 - site incident edit details
Section titled “2024-01-17 - site incident edit details”- Duration: 46h 18m
- Messages: 22
- Resolved: Yes
- Channel:
2024-01-17-war-room-site-incident-edit-details - Summary: likely https://cadence15.sentry.io/issues/4883181944/?project=4505947280572416&query=is%3Aunresolved&referrer=issue-stream&statsPeriod=14d&stream_index=1 this is the place…
Statistics
Section titled “Statistics”- Total incidents: 9
- Total downtime (excluding tracking channels): ~52h (excluding 2024-10-25 tracking channel)
- Average duration (excluding tracking channels): ~6h 30m
- Total messages: 212
- Average messages per incident: 24
- Resolved incidents: 7 (78%)
Note: The 2024-10-25 “gateway timeouts in prod” channel appears to be a tracking channel for ongoing issues rather than a single incident, so its duration (7007h) is excluded from downtime calculations.
Internal & Confidential: This page is only available in the internal handbook and contains confidential information.