Momentum Observability

momentum-observability is an internal-only Fly.io app (one per environment: momentum-observability-{production,staging,development}) that bundles the moving parts of our observability stack into a single deployment. Several processes run side-by-side under supervisord and share a small Starlette dashboard as the front door.

The app has no public IP allocated (fly ips list is empty by design) — *.fly.dev URLs do not work. Reach it via fly proxy or, if you’re on Fly’s WireGuard mesh, http://<app>.flycast:8080.

Access

Forward port 8080 to your laptop and open the dashboard at http://localhost:8080 (HTTP only — there’s no public TLS handler):

# Production
fly proxy 8080:8080 --app momentum-observability-production

# Staging
fly proxy 8080:8080 --app momentum-observability-staging

# Development
fly proxy 8080:8080 --app momentum-observability-development

To bypass the dashboard’s nginx and hit Phoenix directly on port 6006:

fly proxy 6006:6006 --app momentum-observability-<env>

Dashboard

A small Starlette app (dashboard.py) served on port 8080 behind nginx. It’s the landing page for the observability deployment and links out to the three things humans actually look at:

• Phoenix — AI agent traces
• Resource Stats — Fly machine metrics, FPM saturation, autoscaling state
• Postgres — last pgmetrics run, target DB, pg_stat_statements state

The dashboard reads JSON snapshots written to /tmp by the collectors (metrics-latest.json, pgmetrics-status.json) — it doesn’t query Fly or Postgres itself.

Phoenix

Arize Phoenix trace server for our LangGraph agents (LL97, Building Science Expert, Building Guesser). The momentum-calcs service auto-instruments LangChain/LangGraph via OpenTelemetry and exports spans here.

Trace storage uses a Postgres database configured via PHOENIX_SQL_DATABASE_URL. Phoenix runs on port 6006 internally; the dashboard’s nginx proxies it under /projects.

Resource Stats

fly_metrics_collector.py runs every 15 seconds and pulls two kinds of data:

• Fly.io Prometheus API — CPU load, memory usage, instance counts, and HTTP response time percentiles (app-side and edge-side p50/p95) for the discovered apps.
• PHP-FPM status — active/idle/total workers and listen queue depth, scraped in parallel from each machine’s /fpm-status endpoint over the machine’s private IP. FPM is only scraped when autoscaling is enabled (the scrape itself keeps machines awake).

App-side metrics are reported to Sentry as gauges/counters (fly.* and fpm.*) and snapshotted to /tmp/metrics-latest.json for the dashboard. Sentry needs Span Metrics Extraction rules configured before the data becomes queryable.

Environment discovery

The collector requires APP_ENV (production, staging, or development). On every cycle it calls GET /v1/apps?org_slug=cadence-610 and keeps only apps whose name classifies to the deployment’s APP_ENV. Classification is fail-closed — apps that don’t match a rule are excluded.

Pattern	Environment
*-production	production
*-staging	staging
*-development or exactly momentum-main	development
momentum-calcs-pr-* (ephemeral calcs preview deploys)	development
stg-ee-*-momentum (ephemeral preview deploys)	development
*-demo	demo
*-test	test

Apps with status: destroyed are skipped. Newly added or removed apps show up in the next 15s cycle without a redeploy, and changes are logged.

To override discovery — for example, to monitor a single app while debugging — set FLY_METRICS_APPS to a comma-separated list. With it set, classification is bypassed entirely.

Autoscaling

Optional FPM-saturation-driven autoscaling lives in the same metrics collector. Disabled by default; enable with AUTOSCALE_ENABLED=true.

Target apps are not configured separately. The collector autoscales every discovered app for which _is_fpm_app returns true: ephemeral stg-ee-* apps, plus any app whose name starts with momentum and does not contain calcs or observability. So it tracks the metrics-collector’s environment-scoped app list automatically.

Logic in brief:

• Scale up when FPM saturation ≥ AUTOSCALE_SCALE_UP_THRESHOLD (default 0.7), or any requests are sitting in the FPM listen queue, or edge-proxy queueing is detected (Fly edge p95 ≫ app p95).
• Scale down when saturation ≤ AUTOSCALE_SCALE_DOWN_THRESHOLD (default 0.2) and the listen queue is empty. Several extra gates apply: trend must not be rising, no listen-queue blip in the last ~30s, recent p95 latency must have recovered, and the fleet won’t drop below 60% of its peak started count over the last ~2 minutes.
• Cooldowns: 15s after a scale-up, 30s after a scale-down by default. Scale-up cooldown is bypassed on “emergency” signals (queue growing, edge queueing, sat ≥ 90%, or rapid rise).
• The lower fleet bound comes from each app’s min_machines_running in its fly.toml — the collector reads it off the live machine config, not from its own env vars.
• Only starts/stops existing (suspended) machines — never creates new ones. Stops use SIGTERM with a configurable graceful drain (default 120s).

Tunables: AUTOSCALE_SCALE_UP_THRESHOLD, AUTOSCALE_SCALE_DOWN_THRESHOLD, AUTOSCALE_TARGET_SATURATION (default 0.4, used to size scale-up batches), AUTOSCALE_COOLDOWN_UP_SECONDS, AUTOSCALE_COOLDOWN_DOWN_SECONDS, AUTOSCALE_STOP_TIMEOUT_SECONDS, AUTOSCALE_RECOVERY_LATENCY_SECONDS (default 10, blocks scale-down while p95 latency is still recovering from a burst).

pgmetrics

pgmetrics_collector.py runs pgmetrics every 60 seconds (override with PGMETRICS_INTERVAL_SECONDS) and ships the JSON report to pgdash.io via the bundled pgdash CLI. After each run a status snapshot is written to /tmp/pgmetrics-status.json and surfaced on the Postgres tab of the dashboard.

Connection details come from Doppler-supplied env vars — either DB_URL, or per-field DB_HOST / DB_PORT / DB_USER / DB_PASSWORD / DB_NAME / DB_SSLMODE. These are translated to PG* only inside the pgmetrics subprocess; bare PG* vars must not be set on the app, because Phoenix’s asyncpg honors them as fallbacks and breaks against Fly’s internal Postgres.

The collector exits cleanly when PGDASH_API_KEY is unset, which is how environments that don’t ship to pgdash stay quiet.