Data governance and pipelines
A project's data layer is a versioned configuration (sources, elements, mappings, rules, reference lists, targets, access policies) executed by a typed pipeline in the execution engine. The configuration is rows in the project's own database in the versioned configuration store. Every change is a commit with an author, and every run is pinned to a commit.
The node model
The Pipeline page shows the pipeline as a graph of nodes connected by links. Every node comes from one node library: connectors are the load and save node types, transforms are the registered transformation kinds. A node is configured in the canvas and persisted as node and link rows of the recipe.
| Node role | Wraps | Configured by |
|---|---|---|
| Load | A data source: a file, a database, or an agent-driven kind | Upload or connection fields in the node |
| Transform | A transformation kind with its parameters | The kind's own parameter form |
| Save | A data target: a relational or document database | Connection fields in the node |
Underneath, the engine executes data points and transformations. A data point is any named place data lives, with a schema: a source, a target, or an intermediate such as the processed columnar set. A transformation is one typed step from one data point to another. A transformation kind is a registry row (display name, engine endpoint, input and output types, parameter schema, category) backed by exactly one engine endpoint. Two nodes may be linked only when the upstream output type matches the downstream input type. The run executor builds the transformation graph from the node and link rows at run time; the rows are backfilled from existing sources, targets and transformations on startup, so older pipelines keep running.
Data sources
Every source has the same contract: test the connection, discover the schema, sample rows. The schema and an analysis (columns, types, counts, null and distinct statistics from a sample) are stored with the source for the mapping generator. One engine operation dispatches on the source kind, so a new kind is a branch in that dispatcher plus a registry row.
| Family | Kinds | Configured by |
|---|---|---|
| Files | Spreadsheet, CSV, columnar and JSON files | Upload in the Load File node, or an upload followed by ingestion |
| Databases and warehouses | PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, BigQuery, Redshift, MongoDB | Connection fields (host, database, schema or collection, credentials) |
| Object storage | S3-compatible buckets (bucket, prefix or key, region) | Connection fields |
| Message sources | Slack channels, mailboxes | Channel or mailbox, a search query or keyword filter, a date floor, a label, whether thread replies are included, a token |
| External API | Any HTTP API | A master (below), or URL, method, headers and the path to the records in the response |
| Agent-built kinds | Web Scraper, AI Enrichment, News and Feeds, Knowledge Base | A master, from a plain-language brief |
| A new connector type | A registry row whose configuration schema is the form every future connection fills in, bound to an adapter archetype that says how the system is read | The registry |
Dynamic and agent-built sources
Five kinds are built by a master: an engine operation that turns a short plain-language brief into a validated configuration through a fixed sequence of schema-constrained model steps (designed for an inexpensive model) and live probes. The create dialog is brief, Build, verified preview, Create. These sources carry an "AI-built" badge. At run time they are ordinary sources, read afresh on every run.
| Kind | What the master builds | What a run reads |
|---|---|---|
| External API | The request (address, method, headers, paging) and where the records sit in the response, probed by fetching the API | The API, page by page, into rows |
| Web Scraper | A scraping specification for one site or a list of organisations, dry-run before creation | The pages, in "organisations" or "web" mode, into rows |
| News and Feeds | The feed addresses and item fields, probed by reading the feeds | The feeds, one row per item |
| AI Enrichment | The facts to derive per record and the model prompt for each | Each record, calling the configured model provider; every derived fact is unconfirmed with a confidence until a person confirms it |
| Knowledge Base | A document collection: where the documents live in object storage and how they are chunked and indexed | The collection's document rows. Retrieval (semantic search over a per-source vector index) is a separate operation applications and agents call. Each knowledge source stamps an explorer application into the project's Apps |
Message sources have no single schema. Each message or email that passes the filters is an item; a model decides which data elements the item provides and extracts each value, so the mapping is dynamic per item. Extractions keep their history, reads are incremental with a cap per run, and a "load extracted items" transformation pivots them into rows of the target.
A new agent-built family is one master operation on the shared engine plus one catalogue entry. External API, Slack and mailbox kinds share one transport service and support discovery, analysis and reading through the same three operations as databases.
Data elements
A data element is a canonical field of the project's dictionary, with versions. An element version carries data type, definition, owner and status, and may be bound to where it lands in the target (a table and a column, optionally fanned out by a source field). Rules attach to elements with an execution order and run whenever any source maps a column into the element. Overview, Elements, Quality and Data Review in the Data Governance group serve the dictionary.
Source field mappings
A mapping links one source column to one data element, with transformation notes and a confidence between zero and one. "Create Mappings" has the configured model provider read the source analysis and the element dictionary and propose a mapping for every analysed column in one batched call; the source must be analysed and the target elements must exist. Mappings are then reviewed per row on the Source Mappings page or through the governance tools. Mappings and element bindings are internal configuration of the processing-rules kind and never appear as generic node parameters.
Processing rules
A processing rule is a reusable function stored as a row: a name, a type and the rule's code. It receives one cell value (empty and null values are stripped first) and returns whether the value is valid, the transformed value and a message. Rules run inline in the record pipeline, which reads them by name and applies them across all records in element order. Rows a rule rejects land in the invalid set with the rule's message as the reason. There is no endpoint per rule, since rules scale to hundreds across thousands of columns.
Reference lists
A reference list is a controlled vocabulary managed on the Reference Lists page. Rules and elements refer to lists to check that a value is one of the allowed ones.
Targets
A target row names a relational or document database, with the contract test the connection, apply the schema, apply the records. For a relational target the executor applies the schema before the load, then writes the processed records into the tables and columns the element bindings name. The project's first relational target also hosts the workflow tables and the model-prediction table.
Running a pipeline
Run is available from the Pipeline page, the governance tools and the engine's own pipeline endpoints. The recipe's current commit is pinned on the run and a run job is recorded with project, branch and commit. Transformations are sorted topologically by which data point produces each input and dispatched one by one to their kind's engine endpoint through the fork's wrapper, each output feeding the next input. The Runs tab on the Pipeline page lists runs with commit, status and timing.
Registered kinds include processing rules, structured load, extracted load, relational load, document-database load, deduplication, enrichment, cross-field rules and model prediction (see Models and machine learning). The file-to-columnar step is internal and hidden from the library. A new kind is one apply endpoint authored on the fork by the engine persona plus one registry row.
The governance tool surface
The same objects are typed MCP tools on the governance surface, used by the chat's governance persona and by external clients. The read-only surface exposes only the query tools.
| Area | Tools cover |
|---|---|
| Status and navigation | Project status, the suggested next step, navigating the UI, the current project, listing projects |
| Sources | Adding a source, upload and ingestion, analysis, reading analysed columns, querying and updating sources, running enrichment |
| Elements and lists | Creating, updating and querying elements, element details, binding an element to a target, creating and querying reference lists |
| Mappings | Batch proposal, adding one or many, updating, deleting, querying |
| Rules | Creating, updating and querying rules, attaching and detaching rules from elements, attaching the default rules |
| Targets and runs | Adding relational and document targets, updating a target's configuration, running the pipeline, querying runs |
| Branches | Listing, the current branch, the log, the diff, creating, merging, deleting |
| Registry and data points | Registering a transformation kind or a connector type, listing adapter archetypes, listing every data point (intermediates included) with type and schema, assembling the chain for a source and target in one call |
Branch-changing and merge tools ask for confirmation and show a diff first, since a merge into the main branch changes what the next run executes.
Versioning, branches and audit
Every recipe write through the typed tools or the API is a commit authored with the caller's email, and no service writes to the store outside the typed tools. The Branches page and the branch tools create a branch from main, make experimental changes, diff and merge. The branch is part of the page address and of the MCP endpoint address, so a client works on one branch at a time.
Data access policies
Row- and column-level entitlements are part of the recipe and versioned with it.
| Kind | Definition | Effect |
|---|---|---|
| Row filter | A boolean condition over the data's columns with placeholders for attributes of the caller, for example "region equals the caller's region" | The caller's attributes are substituted; an attribute the caller lacks makes the condition false |
| Column mask | A list of columns, optionally with an attribute that lifts the mask | The columns are hidden unless the caller has the attribute |
A policy may apply to one source or the whole project and to some roles; platform administrators bypass. On every data call from an application the API compiles the active policies against the caller's attributes into a short-lived grant (project, expiry, compiled filter, mask, the verified subject) signed with a per-project secret the fork also holds. The engine verifies the signature; an unsigned or invalid grant is ignored and the engine falls back to strict fail-closed compilation. When policies exist an empty grant is still sent for an administrator, so the engine can tell a legitimate bypass from a spoof. A preview shows an organisation administrator what a policy compiles to for a given user.
Where the caller's attributes come from
The subject a policy is evaluated against is assembled server-side on every call, never taken from the browser. It holds the caller's identity (email, organisation, role), the attributes an administrator set on the membership (a tenant id, a region, a list of account ids), and, when the organisation has configured one, the attributes returned by an external attribute provider: the customer's own entitlement or RBAC API, called with the caller's email, with the customer's headers, and read at a chosen path of the response. Answers are cached for a configurable time (five minutes by default). A provider that fails answers no attributes, and a policy that names a missing attribute compiles to false, so an outage narrows access instead of widening it. Tenant isolation is one row filter over the tenant column bound to such an attribute, and it applies to every screen and every agent that reads the governed data. The signed grant carries the whole verified subject, so an endpoint that forwards a call to a customer system can pass on tenant attributes it can trust. Configuration is on the Organizations page (Policy integration). How this fits a customer's existing authorisation is on Integrations and API.
Analytics, charts and reports
The analyst persona reads only the final target. Its MCP surface, reachable only inside the platform, wraps the project's analytics endpoints, authored per project on the fork, and returns charts through its chart tools. An app owner may give an app's viewers this analyst chat instead of the full orchestrator. Reports (Build, Reports) are documents with a mandated form: a template and a query stored as a fork endpoint, rendered by a fork-local render endpoint. A template compile error returns a precise diagnostic, and the manifest's history is versioned like an app's.
Where governance meets applications
Applications never open the database directly. A fleet-built app reads through fork endpoints (read-only queries over the project's target) called through the API's data-call surface, where the signed access grant is attached, so a row filter or column mask applies to every screen. The Pipeline and Governance companions expose these pages inside an app when the owner switches them on. See Applications and Agents and MCP.