Automated Testing Strategy and Glossary
We want to make sure that all our systems are working with automated testing. Why?
- Give us confidence to release more frequently
- Catch issues earlier so they’re easier to fix
- Reduce time spent manually testing
This document lays out our categories of tests. It doesn’t specify the technologies used or details about how the tests are run.
General Guidelines
Section titled “General Guidelines”- Write tests so that they don’t depend on specific UI text that might change, like text or tags. Reliance on those specifics makes tests more brittle.
Unit Tests
Section titled “Unit Tests”Purpose and Scope
Section titled “Purpose and Scope”- Ensure one component works as intended.
- Should cover almost all lines of code, including edge cases and all non-trivial code paths.
- Aim for around 80% code coverage.
- Mock dependencies (e.g., databases or services) since the focus is on testing the component itself.
- Tests should be idempotent to allow parallelization.
- Avoid testing framework or third-party library functionality (e.g.,
MyModel:create()).
Examples
Section titled “Examples”- In Momentum, unit tests already exist in the codebase.
- In the Calcs Library, each notebook has its own tests to validate individual units.
Where They Run
Section titled “Where They Run”- Locally on your machine (great for test-driven development).
- On pull requests to ensure changes don’t break components.
Functional Tests
Section titled “Functional Tests”Purpose and Scope
Section titled “Purpose and Scope”- Ensure multiple units work together in a single application.
- Can test at the service level (e.g., service and database interaction) or UI/API level (e.g., UI component with dependent services and data-layer components).
- Mock external APIs.
- Aim for 50% code coverage (not every branch of every conditional needs coverage).
- Tests should be idempotent so a new test won’t break an existing test, and so test execution order doesn’t matter.
- Useful for covering refactors where underlying components are reorganized.
Examples
Section titled “Examples”- In Calc Service, API layer tests are functional tests.
- In Momentum, feature tests ensure livewire components load and function correctly while relying on services and data models.
Where They Run
Section titled “Where They Run”- Locally on your machine.
- On pull requests.
- Do not run on deployed environments (e.g., staging or production) as they manipulate data.
End-to-End Tests
Section titled “End-to-End Tests”Purpose and Scope
Section titled “Purpose and Scope”- Ensure all apps work together (e.g., Momentum, Calc Service, BKB, and third-party components with sandbox environments).
- Test the happy path for expected use cases and some common unhappy paths.
- Tests are not idempotent and may have interdependencies.
- Aim for 30% code coverage (tests are fragile and break with UI changes).
Examples
Section titled “Examples”- Currently (as of March 2025), these tests are performed manually by Jeff as regression test scripts.
- Future automation may involve a browser driver manipulating the web client through various use cases.
Where They Run
Section titled “Where They Run”- On staging (or a dedicated environment) after a branch is merged.
Smoke Tests
Section titled “Smoke Tests”Purpose and Scope
Section titled “Purpose and Scope”- Ensure critical user flows are working in production.
- Detect deployment or configuration issues (e.g., mismatched environment variables, incorrect
.tomlsettings, incompatible third-party API changes). - Unlikely to find bugs but ensure users can perform central tasks.
Examples
Section titled “Examples”- Currently (as of March 2025), these tests are performed manually as a 5-minute script after each release.
- There is a proposal to automate this process.
Where They Run
Section titled “Where They Run”- Ideally triggered in production after an automated deploy.
- Report issues to a high-priority Slack channel as an outage.