Tests and health
Every requirement in Docs, Requirements can carry any number of tests. A test is a declarative specification, never code. It names a target (the project's endpoints, its relational target database, its application, an agent) and never a host. One portable runner executes a project's test manifest inside cortex, inside every archive the project is delivered as, on the agent fleet and on any machine. The verdicts fold into one health verdict per project, shown on the Status page and announced on transitions.
The format
A test manifest names the project and lists tests. Every test has a stable key (unique per project; results are matched on it across environments), the requirement it proves, a name, a kind, the specification for that kind, and an enabled flag.
| Kind | What it runs | What the assertions read |
|---|---|---|
| Endpoint | One of the project's engine endpoints with an input (in cortex through the fork; in an archive through the bundled endpoints runtime) | Whether the call succeeded, its payload and its duration |
| HTTP | A request on a named target with a path or URL, a method, headers and a body; with no assertions a successful status passes | The status, the headers, the body and the duration |
| SQL | A query with parameters on the relational target database | The rows, the row count and the duration |
| Browser | A sequence of steps that drives the application in a headless browser, with a viewport and a rule for console errors (record, fail or ignore) | The final URL and title, each step's outcome, the values read during the run, the screenshots and the console errors |
| Agent | One question to one of the project's agents, asked through the evaluation surface exactly as the chat would, one model run per test | The answer, whether it was refused, the safeguard verdict with its reason and checks, the tool calls and their count, the duration |
An assertion names a path into the subject, an operator and, where needed, a value. Operators: equals and not equals; greater than, at least, less than, at most; exists and missing; truthy and falsy; contains and does not contain; matches or does not match a pattern; has a type; has a length, minimum length or maximum length; is one of a list. An endpoint, HTTP or SQL test may allow errors, so a thrown call becomes a subject recording the failure and a refusal can be asserted. Verdicts per test: passed, failed, error (the test could not run), skipped.
Targets, fixtures and secrets
Targets are resolved by an environment description, never by the specification: the project's endpoints, its relational target database, the application, the portal API, the dashboard, the agents, and one target per app service. A target the environment lacks makes the test skipped, never failed. Cortex fills the environment from the project's fork and target; an archive bakes its own at package time.
A specification may require labels the environment must provide: sample data seeded for a trial, a named development stand-in, or the operations layer (the app SDK and the API proxy switched on). Cortex provides everything; a missing label skips the test with the label as the reason. A specification may also read a credential from a target's environment through a placeholder naming the target and the key (in cortex, the app service's environment settings; in an archive, the service's own environment file). An unset value skips the test. A specification never carries a literal secret.
Browser steps
Steps are data. Each may carry its own timeout, be marked optional, and name the frame it acts in when the application is embedded. The steps: open a page (the application's view, a path, a URL, or a path on a named app service, optionally waiting for a readiness condition); click and hover; fill a field and select an option; press a key; wait for an element to reach a state; wait for or expect a text (exact or by pattern, optionally within an element); expect a count of elements; expect an element visible or hidden; expect the URL to contain or match; read a text or a count into a named captured value; take a screenshot (of an element or the full page); sleep, at most ten seconds.
An element is addressed by a selector string or by a role with an optional name, a test id, a text, a label, a placeholder or a CSS selector. Text checks are case-insensitive and whitespace-tolerant. The first failing expectation fails the test with that step as the message; an action that cannot run is an error. Screenshots are PNG files, filed as evidence on the requirement's tasks when a run is imported into cortex (see Requirements and knowledge).
Where tests run
| Where | How | Results |
|---|---|---|
| The cortex backend | Starts the runner as a child process against the project's fork and its first relational target; no browser there, so browser tests report skipped | Runs and results in the platform database |
| The agent fleet | Runs the browser kind with its own headless browser and imports the run into cortex; a no-model lane does this for the schedule | Imported runs, screenshots as evidence |
| Every archive | A tests service with a headless browser and a script to run on demand; the bundled portal proxies the results so the archive's own Status page shows its runs | Results on a volume, importable back into cortex |
| Any machine | The runner with a manifest and an environment description; a non-zero exit when anything failed | A results document on standard output |
A skipped verdict never outranks a real one. The Status page shows each test's newest non-skipped verdict across runs, so a fleet browser pass and a backend endpoint pass compose into one answer.
Authoring
Tests are authored on the Tests section of each requirement. Write tests books a fleet generation from the requirement's use cases and the project's real surfaces (endpoints, tables, screens). The platform validates every returned specification with the runner's own validator before a row is written; Add test writes one by hand under the same validator.
A generated test that lands red books one review per requirement. The reviewer sees the requirement, the red specification, the failed step and the application's visible text at that moment, and answers update, keep or disable, with a reason. A test is reviewed at most twice, then stays red for a person. Watchdog runs never book the reviewer.
The rules: one requirement per test, derived from its use cases; read-only; portable (a target and a path, required labels, placeholders); stable keys; small and independent. Every builder skill of the fleet ends by authoring and running tests for the requirements it made true. A browser test is the UI verification and replaces browser-automation scripts kept in a repository.
The Status page and the health verdict
Status (Operations group) shows the latest run, every test with its newest verdict grouped under the requirement it proves, the run history and Run all. The same page reads a deployed archive's own health through the bundled portal.
The platform folds the newest non-skipped result per enabled test into one verdict per project: green (no failing test and at least one passed), red (any failed or errored test), none. Transitions are: went red, went green, the failing set changed while red, and crashed (the runner died without verdicts). Only transitions are announced, so a red project does not page every fifteen minutes.
The watchdog
The platform runs every project's enabled tests every fifteen minutes by default (an environment setting), once about two minutes after the backend starts, and about four minutes after a deploy starts. The schedule can be switched off per environment; on-demand and gate runs stay. Browser tests for the schedule are booked as jobs for the fleet. Agent tests run on demand and after a deploy, and on the schedule only where the environment opts in, because each is a model run.
A transition files an Alert in the project's Alerts feed and posts to Slack through an incoming webhook, prefixed with the environment's name so one channel can carry several environments. The message names the failing tests and links the Status page. See Security and operations.
The push gate
Store parity is not application health. A project copy pushed to a cluster environment finishes with a test gate that starts the suite on the target and succeeds only when every non-skipped test passed and at least one ran. It fails when the suite is red, and fails differently when the run could not start or finish in time. The push refuses a red copy. See Deployments and environments.
Each run records what triggered it: manual, push, deploy, release, schedule, startup, or a generation.