Database Health & Recovery

Momentum and Calc Service rely on two self-managed Fly Postgres databases:

Database	Typical Production App Name
Application DB	application-db-production
Building Knowledgebase (BKB) DB	bkb-prd-* (versioned, e.g. bkb-prd-robinneri-momentum-bkb-v2-2-0-20260621-231819)

Both databases are regular Fly Postgres apps and use the same health check and recovery procedures.

App names can change

The Application DB is occasionally rebuilt or restored from backup, which gives it a new app name. The BKB DB is always versioned. Use the Calc Service debug page (below) to confirm the current names before running any commands.

App-Side Status Pages

These pages show live database connectivity, including the current app names for both databases.

Service	URL	Access
Momentum (PHP)	https://momentum.c15.io/admin/diagnostics	Public (admin login required)
Calc Service	http://momentum-calcs-production.flycast:8080/debug/status	VPN / fly proxy only

The Calc Service page is only reachable over Fly’s WireGuard VPN — .flycast addresses do not work from a browser without VPN. If you’re not on VPN:

8080/debug/status

fly proxy 8080:8080 --app momentum-calcs-production

Check Health

Use the fly-machines.py script (Momentum repo, scripts/fly-machines.py) for a quick at-a-glance view of machines, health check status, and resource utilization:

./scripts/fly-machines.py <app-name>

This is the fastest way to spot a stopped or zombie machine. See Capacity & Scaling for a full description of the output columns.

For more detail on individual health checks:

# List health checks for all machines
fly checks list -a <app-name>

# Checks per machine:
#   pg   — Postgres health (should be "passing")
#   role — Machine role (should be "primary" or "replica", not "zombie")

# Inspect a specific machine
fly machine status <machine-id> -a <app-name>

A healthy cluster looks like:

NAME │ STATUS  │ MACHINE        │ OUTPUT
─────┼─────────┼────────────────┼───────
pg   │ passing │ 7846415a6e67d8 │ 200 OK
role │ passing │ 7846415a6e67d8 │ primary
pg   │ passing │ 9876543a1b2c3d │ 200 OK
role │ passing │ 9876543a1b2c3d │ replica

Symptoms of an Unhealthy Machine

• pg check shows critical with a timeout error
• role check shows zombie instead of primary or replica
• Sentry reports SQLAlchemy error: connection refused to the database’s internal hostname (see momentum-calcs#2027 for a BKB example)

Recovery: Zombie Machine

1. Try a restart first:

fly machine restart <machine-id> -a <app-name>

2. If restart fails — destroy and clone from a healthy replica:

# Destroy the zombie machine (--force skips confirmation)
fly machine destroy <machine-id> -a <app-name> --force

# Clone from a healthy machine (find its ID from fly checks list)
fly machine clone <healthy-machine-id> -a <app-name>

3. Verify recovery:

fly checks list -a <app-name>

All machines should show pg: passing and a valid role (primary or replica).

Related Resources

• Capacity & Scaling — scaling procedures for Fly apps and databases
• Momentum Observability — pgmetrics and pgdash access
• Backup Policy — snapshot and recovery procedures
• Incident Response Plan — escalation paths