Infrastructure Knowledge from Team Discussions

This document captures infrastructure, deployment, security, and DevOps practices that emerged from team discussions. This information should be integrated with existing infrastructure documentation.

Environment Configuration

Environment Variables

Configuration management:

Use getenv() for environment variables (not ENV->config in some cases)
.env.testing should be in .gitignore (not committed)
.env.testing.example should exist as template
Convention: Have .env.example and .env.testing.example only; ignore .env and .env.testing

Doppler integration:

Environment variables managed via Doppler
Changes in Doppler should be picked up by applications
Testing Doppler functionality falls under end-to-end or smoke tests
Debug panel can show values to verify Doppler integration

Database Configuration

Staging vs Demo environments:

Both derived from anonymized snapshots of production
Demo tied to versions equivalent to prod (separate from staging due to mutations not yet in prod)
Both hosted on cheap hardware (single machine, no redundancy, shared, medium specs) at ~$11/mo
Staging/demo databases not typically suspended due to low cost
Momentum and calc service apps suspended (on performance hardware)

Database access:

Read-only credentials needed for analysis work
Helps run analysis more safely without risk of accidental changes
Should be created as part of infrastructure setup

API Key Management

Google Cloud API Keys

Previous issues:

Using unrestricted API keys
Shared keys across prod/staging/dev
Unclear naming in console

Current approach:

Created separate, restricted API keys for different environments
Local dev key: Restricted to subset of APIs actually used
Key location: See Doppler dev config (key ends in ...6FY*g instead of ...GhzY)
For calc service: Use same key in GOOGLE_API_KEY

Security considerations:

Restricted APIs but not domains (leaves gap since Street View URLs include API key)
Plan: Leave dev key unrestricted on URLs; tighten staging and prod
Note: Google’s terms prohibit caching Street View images (with rare exceptions), so can’t proxy to protect user data

Best practice (from team discussion):

For production apps, use local proxy: Backend loads image with secret API key and stores locally
Protects user data from being leaked to Google
May be overkill but quick to implement

API Key Restrictions

Domain restrictions:

Local dev: May need to allow momentum.test domain
Staging/prod: Should have tighter domain restrictions
Balance between usability and security

Deployment and Environments

Fly.io Deployment

Applications:

Momentum and calc service deployed on Fly.io
Performance hardware (can be suspended when not needed)
Staging and demo databases on cheaper hardware

Deployment process:

PRs to main deploy to dev environment
PRs to release deploy to staging/release environment
Production deployments from release branch

Environment Purposes

Development (momentum-development):

Deployed from main branch
Never locked
For active development and testing

Staging (momentum-staging):

Deployed from release branch
Always locked
For release candidate testing

Demo:

Tied to production-equivalent versions
Used for client demos
Available outside VPN (like prod)
Also used for external vulnerability testing (e.g., NYCA)

Production:

Deployed from release branch
Final testing before production deployment

Database Management

Database Snapshots and Backups

Backup practices:

Create backups before large database changes (e.g., SQL statements)
Manual database changes should be coordinated
Pair on large changes when possible

Anonymized snapshots:

Used for staging and demo environments
Need to be updated periodically
May contain invalid data that causes issues (e.g., SSO registration problems)
Process needed for getting more recent anonymized dumps

Database Migrations

Migration practices:

Migrations should only work with schema changes
Updating data because of schema changes is acceptable
Don’t use migrations for general data updates

Database setup:

Use scripts/setup_db.sh for database setup
Use --testing flag to create test databases and users
Script creates database and user properly configured

Monitoring and Error Tracking

Sentry Configuration

Current setup:

Sentry SDK present and configured
Possible issues:
- Sentry.js configured to use “production” session, not “staging” or “development”
- Client (browser) errors caught by Livewire, sent to Sentry using backend (Laravel) configuration
- Errors sent to momentum project regardless of environment

Configuration needs:

Proper environment detection
Separate projects or proper tagging for staging/dev
Review how Sentry is booted and configured in codebase

Security Practices

SSO and Authentication

Microsoft SSO:

Testing on staging environment
Well-known endpoint: /.well-known/microsoft-identity-association.json
Needs to be unhidden/enabled for testing

SSO registration issues:

May be caused by invalid data in anonymized database dumps
Need process for updating or fixing database dumps
May require generating random non-nullable passwords for users

Access Control

Read-only database access:

Needed for analysis work
Helps prevent accidental changes
Should be created as part of infrastructure

API access:

Restrict API keys to necessary APIs only
Use domain restrictions where possible
Separate keys for different environments

Infrastructure Improvements

Container Updates

Docker container updates:

Automatic Monday container updates
Merged post-release as part of deploy process
Can be auto-merged (CodeRabbit may label as review skipped, which is OK)

Docker Compose updates:

Updating to newer versions (e.g., 8.5) can improve performance
Sail runs tests faster with updates
Need to coordinate updates to avoid breaking local setups

Performance Optimization

Caching strategies:

For computation-heavy services (e.g., calc-service), caching may make sense
Short cache with cache tags for invalidation
Tricky when having two different services, but can work

Resource management:

Suspend performance hardware when not needed
Keep staging/demo on cheap hardware (low cost, not worth suspending)
Balance cost and availability needs

Development Tools

PostHog Setup

Configuration:

Create PostHog account (personal or team)
Get POSTHOG_API_KEY from PostHog
Put API key in Doppler
Set POSTHOG_HOST=https://app.posthog.com
Can share credentials for team access

Environment-Specific Tools

Local development:

Some developers use Herd or phpmon instead of Docker
Important to communicate setup changes
Document alternative setups

This document is a synthesis of team discussions. For official infrastructure documentation, see the main DevOps and Infrastructure documentation.