Capacity & Scaling Resources

Document Status

Draft — Skeleton Only This document is a starting point. Resource specs, app names, and specific thresholds need to be filled in by the team. See momentum#12013 for the tracking issue.

Overview

This runbook provides step-by-step procedures for emergency upscaling of Cadence OneFive’s Fly.io applications and databases when load spikes or incidents occur. Use this when:

• Monitoring alerts indicate elevated latency or error rates
• A known traffic spike is expected (e.g., client demos, load testing)
• An incident is in progress and additional capacity is needed

Note

For load testing context and the broader scaling plan, see momentum#9617.

Checking Status of a Fly App

Use the fly-machines.py script in the Momentum repo to quickly check the status of any Fly app:

./scripts/fly-machines.py <app-name>

Example output for momentum-production:

                                  momentum-production  (2 started, 2 stopped)
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━┳━━━━━┳━━━━━┳━━━━━━┳━━━━━━━━━━━━━━┳━━━━━┳━━━━━━━━━┳━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━┓
┃Name                     ┃ St ┃ Grp ┃ Rgn ┃  CPU ┃ Mem          ┃ Chk ┃ Deploy  ┃ Upd ┃ Size ┃ ID            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━╇━━━━━╇━━━━━╇━━━━━━╇━━━━━━━━━━━━━━╇━━━━━╇━━━━━━━━━╇━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━┩
│empty-wave-3218          │ 🟢 │ app │ iad │ 1.08 │   30% ██░░░░ │ 1/1 │ 01KMS18 │  4d │ p2x4 │ 4d893d55a20187│
│billowing-resonance-1964 │ 🟢 │ cro │ iad │ 0.00 │    9% █░░░░░ │  —  │ 01KMS18 │  4d │ p2x4 │ 3d8d2d69b04389│
│solitary-bird-2276       │ 🔴 │ bkf │ iad │    — │ —            │  —  │ 01KMS18 │ 22h │ s1x0 │ 148e0299a15789│
│dawn-night-4261          │ 🔴 │ app │ iad │    — │ —            │  —  │ 01KMS18 │ 20h │ p2x4 │ 148e0295f19589│
└─────────────────────────┴────┴─────┴─────┴──────┴──────────────┴─────┴─────────┴─────┴──────┴───────────────┘

Key columns:

• St — machine state (🟢 started, 🔴 stopped)
• Grp — machine group (app, cro=cron, bkf=backfill, rel=release)
• Size — VM size (p=performance, s=shared; e.g., p2x4 = performance-2x with 4GB)
• Chk — health check status
• Mem/CPU — current utilization

This is the quickest way to assess the current state of an app before making scaling decisions.

How Fly Autoscaling Works

Fly will automatically start and stop machines based on traffic, but only up to the number of machines you have allocated for the app. In the momentum-production example above, there are 4 machines total (2 app, 1 cro, 1 bkf) — Fly will never spin up a 3rd app machine on its own.

Auto-suspend keeps idle machines free

There is no downside to allocating more machines than you expect to need, because our fly.toml configs use auto_stop_machines = "suspend". Suspended machines cost nothing (no CPU billing) and wake automatically when traffic arrives.

From momentum-production.toml:

[http_service]
auto_stop_machines = "suspend"
auto_start_machines = true
min_machines_running = 1

From fly-calcs-production.toml (also includes concurrency-based scaling triggers):

[[services]]
auto_stop_machines = "suspend"
auto_start_machines = true
min_machines_running = 1  # Keep at least 1 warm to avoid cold starts

[services.concurrency]
  type = "requests"
  soft_limit = 3    # trigger scale-out at this concurrency
  hard_limit = 20   # absorb bursts while scaling out

Increasing machine count with fly scale

To give Fly more headroom to autoscale, increase the machine count:

# Check current machine count (see "Checking Status" above)
./scripts/fly-machines.py <app-name>

# Scale to N machines for a process group (e.g., give the app group 4 machines)
fly scale count app=4 -a <app-name>

# Scale multiple groups at once
fly scale count app=4 cron=1 -a <app-name>

Because suspended machines are free, you can proactively over-provision. For example, scaling momentum-production from 2 to 6 app machines means Fly can handle a 3x traffic spike without manual intervention — and when traffic drops, the extra machines suspend automatically.

Current Resource Specs

Momentum (PHP)

App	Environment	VM	Region	Min Running
momentum-production	Production	perf-2x / 4 GB	iad	1
momentum-staging	Staging	perf-2x / 4 GB	iad	0
momentum-demo	Demo	perf-2x / 4 GB	iad	0
momentum-main	Development	perf-2x / 4 GB	iad	0

Production also runs cron and backfill-worker process groups. Backfill workers use shared-1x (512 MB production, 256 MB elsewhere).

Calcs Service (Python)

App	Environment	VM	Region	Min Running
momentum-calcs-production	Production	perf-2x / 4 GB	iad	1
momentum-calcs-staging	Staging	perf-2x / 4 GB	iad	0
momentum-calcs-demo	Demo	perf-2x / 4 GB	iad	0
momentum-calcs-development	Development	perf-2x / 4 GB	iad	0

Concurrency limits: soft=3, hard=20 (request-based).

Databases (Fly Managed Postgres / MPG)

Cluster Name	ID	Environment	Plan	VM	Region
application-db-production	nlkxjo59x54ry93v	Production	Launch	perf-2x / 8 GB	iad
application-db-staging-managed-anonymized-2025-07-09	zp2wjre9e450dn4q	Staging	Starter	shared-2x / 2 GB	iad
application-db-testing-managed-anonymized-2025-07-09	82ylg0166z3rzx19	Testing	Basic	shared-2x / 1 GB	iad

MPG plans:

Plan	CPU	Memory	Price
Basic	shared-2x	1 GB	$38/mo
Starter	shared-2x	2 GB	$72/mo
Launch	perf-2x	8 GB	$282/mo
Scale	perf-4x	32 GB	$962/mo
Performance	perf-8x	64 GB	$1,922/mo

Emergency Vertical Scaling (CPU/Memory)

Vertical scaling increases the resources available to existing machines.

Generic Applications

# 1. Check current machine specs
./scripts/fly-machines.py <app-name>

# 2. Update machine size (e.g., from shared-cpu-1x to performance-2x)
fly machines update <machine-id> --vm-size <new-size> -a <app-name>

# 3. Verify the change
./scripts/fly-machines.py <app-name>

Available VM sizes (verify against current Fly.io docs):

Size	CPU	Memory
shared-cpu-1x	1 shared	256 MB
shared-cpu-2x	2 shared	512 MB
performance-1x	1 dedicated	2 GB
performance-2x	2 dedicated	4 GB
performance-4x	4 dedicated	8 GB
performance-8x	8 dedicated	16 GB

MPG Databases

MPG databases scale by changing plans, not by resizing individual machines. Read replicas and autoscaling are not yet available.

Changing MPG tiers causes downtime

Changing an MPG plan tier (e.g. from Launch to Scale) causes service disruption. During the transition, request failure rates can spike to 50%+ for several minutes:

Tier changes are done through the Fly.io dashboard (Settings tab for the MPG cluster), not via the CLI.

Right-size the MPG tier before you need it. Don’t plan to change tiers reactively during a load spike — the downtime makes a bad situation worse. If you anticipate needing more database capacity, upgrade the tier during a maintenance window well in advance.

# List MPG clusters and their IDs
fly mpg list

# Check status of a specific cluster
fly mpg status <cluster-id>

Capacity and Bottlenecks

Based on load testing against staging (2026-03-31, momentum#12289):

Momentum (PHP) is the bottleneck. Scale it first. The app has poor per-machine throughput due to N+1 queries, expensive Blade rendering, and heavy model hydration. Tickets filed to address these (#13366–#13376), but until they’re resolved, throwing machines at it is the primary lever.

The database and calcs service have significantly more headroom:

• Application DB handled 30 app servers at 1,000 queries/second without stress. Don’t be afraid to run 20 momentum machines against it.
• momentum-calcs benefits from momentum-side caching and is relatively light on resources. Keep an eye on capacity, but it’s unlikely to be the constraint. One known hot path is BatchedScopeSummaries (1.4s avg, CPU-bound — CadenceOneFive/momentum-calcs#1496), and p95 latency doubles under contention on team_building_attributes (CadenceOneFive/momentum-calcs#1497).

What to do in a load spike:

1. Scale momentum app machines first (fly scale count app=N)
2. Monitor calcs — if concurrency soft limit (3) is being hit consistently, add machines
3. DB is unlikely to need intervention at current scale

Use fly-machines.py --env prod to watch all production machines across the cluster at once. Load can distribute unevenly across momentum machines — monitor per-machine CPU after scaling.

Related Resources

• Database Health & Recovery — checking health and recovering from failures for the Application DB and BKB DB
• SLOs & Monitoring — incident severity levels and response framework
• Fly.io Scale-to-Zero — the inverse operation (scaling down idle environments)
• Fly.io Scaling Documentation — official Fly.io docs
• Fly.io Postgres Scaling — official Postgres scaling docs