Data Mesh Architecture: Principles, Components, and Data Products as Code

TL;DR — Data mesh is an operating model in which domain teams own analytical data as products, built on a shared self-serve platform and governed by rules that platform enforces automatically. It works when each data product is versioned code with an owner, a contract, an SLA, and quality checks, and when an orchestration layer gives every domain its own namespace while the platform team applies governance centrally. It is not a storage technology, and it is overkill for an organization with one data team.

Data mesh architecture is a decentralized approach to analytical data in which business domains own, build, and serve their data as products, on top of a shared self-serve platform and under federated governance. Zhamak Dehghani introduced the idea in a 2019 article on martinfowler.com and formalized it as four principles in December 2020.

The problem data mesh targets is organizational. A central data team that owns every pipeline becomes the queue that every analytics request waits in, and the people modeling the data sit furthest from the business logic. Data mesh moves ownership to the teams that produce the data (sales, payments, logistics) and turns the central team into a platform team that builds the paved road.

The definitions are the easy part. Most programs stall at implementation: what exactly is a data product in the repository, who runs it, and how does a rule get enforced across dozens of teams without a review board? This guide covers the principles and the components, then the practitioner layer: data products as versioned code, an orchestrator as the self-serve platform, and governance written as policies.

The four principles of data mesh, and what each one changes

The four principles of data mesh are domain-oriented ownership, data as a product, self-serve data infrastructure as a platform, and federated computational governance. Each one removes a specific failure mode of centralized data platforms, and each one only holds if the next one is in place.

1. Domain-oriented ownership

Analytical data is owned by the business domain that generates it or understands it best. The orders domain owns its orders data end to end: extraction from the order service, transformation, tests, documentation, and the on-call rotation when a load breaks. Dehghani distinguishes source-aligned data (close to operational systems), aggregates, and consumer-aligned data shaped for a specific use.

What changes in practice: pipeline work moves into the domain’s backlog and the domain’s on-call. Domain boundaries should follow the boundaries already used in the software architecture, not the org chart of the old data team.

2. Data as a product

Domain data is published for other teams with the discipline of a public API: discoverable, addressable, trustworthy, self-describing, interoperable, and secure. Dehghani calls the data product the “architectural quantum” of the mesh, the smallest unit that can be deployed independently and still carries everything it needs: code, data and metadata, and the infrastructure to run it.

What changes in practice: every product has a named owner, a published schema, a freshness SLA, quality checks that run before publication, and a version. A table in a shared schema without those properties is a dataset, not a data product.

3. Self-serve data infrastructure as a platform

A platform team provides the infrastructure, tooling, and templates domains need to build products without filing tickets. Dehghani describes three planes: an infrastructure provisioning plane (storage, compute, access), a data product developer experience plane (build, test, deploy, and monitor a product), and a mesh supervision plane (discovery, lineage, and global policies across products).

What changes in practice: the platform is judged by how quickly a new domain ships its first product without the platform team writing that product’s code.

4. Federated computational governance

Domain representatives and platform owners jointly define global rules (identifiers, classification, access, retention, interoperability standards), and the platform enforces them automatically. “Federated” describes who decides. “Computational” describes how rules apply: as code the platform evaluates, not as a meeting that approves changes.

What changes in practice: a data product without an owner label, or one that breaks the classification policy, fails at deploy time instead of being caught in a quarterly audit.

Data mesh architecture components and how they fit together

A data mesh architecture has four layers: domains that own data products, the data products themselves with input and output ports, a self-serve platform shared by every domain, and a governance layer encoded into that platform. Dependencies run from product to product, not through a central team.

+--------------------------------------------------------------------------+
| GOVERNANCE (federated, computational)                                    |
| global rules as code: ownership, classification, SLAs, access, naming    |
| defined by a cross-domain group, enforced by the platform on every save  |
+--------------------------------------------------------------------------+
| DOMAINS AND DATA PRODUCTS                                                |
|                                                                          |
|  Sales domain             Payments domain          Finance domain        |
|  orders_daily             refunds_daily            revenue_daily         |
|  (source-aligned)         (source-aligned)         (consumer-aligned)    |
|  contract, owner,         contract, owner,         contract, owner,      |
|  SLA, checks              SLA, checks              SLA, checks           |
|       |                        |                         ^               |
|       +---- output port -------+------ output port ------+               |
+--------------------------------------------------------------------------+
| SELF-SERVE PLATFORM (owned by the platform team)                         |
| provisioning plane: schemas, storage, compute, secrets                   |
| developer experience plane: orchestration, templates, CI, testing        |
| supervision plane: catalog, lineage, quality and SLA monitoring          |
+--------------------------------------------------------------------------+
| SHARED INFRASTRUCTURE: object storage, warehouse or lakehouse engines    |
+--------------------------------------------------------------------------+

Each component has a concrete counterpart in a modern data stack:

Nothing in this table requires a specific storage engine. Many meshes run on one warehouse, or on a single lakehouse architecture, with a schema or catalog per domain. Decentralization concerns ownership and deployment, not physical data placement. The catalog layer is where consumers find products and see how they connect; data catalog lineage explains why that graph is only trustworthy when execution feeds it rather than hand-maintained documentation.

A data product is code, data, and a contract under version control

A data product becomes manageable at scale when everything that defines it lives in one versioned repository: the transformation code, the contract consumers rely on, the ownership metadata, the SLA, the quality checks, and the trigger that runs it. That is what “data as a product” means operationally. If the definition is split across a wiki page, a catalog form, a cron entry, and a dashboard, nobody can review, roll back, or audit the product as one thing.

A typical repository layout for one product:

data-products/sales/orders_daily/
  dataproduct.yaml            # catalog entry: id, domain, owners, output assets
  contract.yaml               # ODCS contract: schema, quality rules, SLA, support channel
  dbt/models/orders_daily.sql
  dbt/models/orders_daily.yml # dbt tests and model contract
  flows/orders_daily.yaml     # Kestra flow: trigger, build, checks, SLA, publication

Two open specifications make the contract and catalog parts concrete. The Open Data Contract Standard (ODCS), maintained by the Bitol project, defines a YAML data contract with sections for schema, data quality, team, roles, support channels, and service-level agreements. On the catalog side, DataHub supports declaring data products in YAML managed with GitOps practices and synced with datahub dataproduct upsert -f. A minimal DataHub data product file looks like this:

id: orders_daily
domain: Sales
display_name: Orders (daily)
description: One row per order and day, settled amounts in EUR.
assets:
  - urn:li:dataset:(urn:li:dataPlatform:postgres,analytics.sales.orders_daily,PROD)
owners:
  - id: urn:li:corpGroup:sales-data
    type: TECHNICAL_OWNER
properties:
  freshness_sla: "published by 06:15 UTC"

The catalog file declares what the product is. It does not run anything. Something still has to build the product on time, refuse to publish data that breaks the contract, and tell downstream domains that a new version is ready. That is the job of the orchestration layer, and it is also where most of the data quality guarantees become enforceable rather than aspirational.

How an orchestrator becomes the self-serve platform layer

In a data mesh, the orchestrator is the part of the self-serve platform that turns a product definition into scheduled, observed, and governed executions. It has to give each domain autonomy over its own pipelines while giving the platform team one place to apply standards. That split is the core requirement of data orchestration in a mesh, and Kestra maps it onto five mechanisms.

Per-domain namespaces. Kestra namespaces are hierarchical (acme.sales.orders, acme.finance.revenue), and every flow belongs to exactly one. A namespace groups the domain’s flows, its namespace files (dbt projects, SQL, scripts), and its KV store entries. In the Enterprise Edition, namespaces also scope secrets, variables, RBAC role bindings, and audit logs, so the sales team deploys and runs flows in acme.sales without any access to finance.

Flows owned by domain teams. A flow is a YAML file the domain team writes, reviews, and deploys from its own Git repository. The platform team does not approve each change; it defines what every flow must satisfy and lets the platform check it.

Shared subflows and templates. The platform team publishes reusable steps as subflows (record freshness, register in the catalog, notify consumers) that domains call with the io.kestra.plugin.core.flow.Subflow task instead of copying the logic. Custom Blueprints, which let the platform team publish templated flow skeletons that teams instantiate from a form, are an Enterprise Edition feature.

Cross-domain triggers. Consumer domains depend on producer products through events, not through guessed cron times. The io.kestra.plugin.core.trigger.Flow trigger starts a flow when upstream flows in other namespaces reach a given state, with dependsOn conditions, an ALL, ANY, or AT_LEAST mode, and an optional deadline window. Finance does not need to know when sales runs its build; it reacts when the orders product publishes.

Central governance. Policies (Enterprise Edition) are evaluated when a flow is saved and before it executes. They inject values, require properties such as an owner label, restrict values, or deny plugins, and child namespaces inherit parent rules and can only tighten them. Asset lineage, also Enterprise Edition, records which flows read and write which tables, so the supervision plane comes from execution metadata instead of a diagram someone forgot to update.

Example: the sales domain’s orders data product in YAML

The flow below is a complete data product pipeline owned by a sales domain team. It builds orders_daily with dbt into an audit schema, measures the contract with SQL, blocks publication if any guarantee fails, swaps the audited data into the published schema in one transaction, and hands off to a platform-owned publication subflow. The pattern is write-audit-publish: consumers never see a build that failed its checks.

id: orders_daily
namespace: acme.sales.orders

description: |
  Data product sales.orders_daily, owned by the sales-data team.
  Contract: one row per order, order_id unique, amount_eur >= 0.
  Output port: analytics.sales.orders_daily, published by 06:15 UTC.

labels:
  data_product: orders_daily
  domain: sales
  owner: sales-data
  contract_version: "1.4.0"

# Runtime SLA: the build starts at 05:30, so failing after 45 minutes
# keeps the 06:15 publication promise visible instead of silently late.
sla:
  - id: publish_window
    type: MAX_DURATION
    duration: PT45M
    behavior: FAIL
    labels:
      sla: miss

tasks:
  # Write: dbt builds the models and runs dbt tests in the audit schema.
  - id: build
    type: io.kestra.plugin.dbt.cli.DbtCLI
    containerImage: ghcr.io/kestra-io/dbt-postgres:latest
    namespaceFiles:
      enabled: true # dbt project synced from the domain's Git repository
    commands:
      - dbt deps
      - dbt build --select +orders_daily --target prod
    profiles: |
      sales:
        outputs:
          prod:
            type: postgres
            host: "{{ secret('WAREHOUSE_HOST') }}"
            port: 5432
            user: "{{ secret('WAREHOUSE_USER') }}"
            password: "{{ secret('WAREHOUSE_PASSWORD') }}"
            dbname: analytics
            schema: sales_audit
            threads: 4
        target: prod

  # Audit: measure the contract guarantees on the freshly built table.
  - id: contract_metrics
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://{{ secret('WAREHOUSE_HOST') }}:5432/analytics"
    username: "{{ secret('WAREHOUSE_USER') }}"
    password: "{{ secret('WAREHOUSE_PASSWORD') }}"
    fetchType: FETCH_ONE
    sql: |
      SELECT
        COUNT(*) AS row_count,
        COUNT(*) - COUNT(DISTINCT order_id) AS duplicate_ids,
        COUNT(*) FILTER (WHERE amount_eur < 0) AS negative_amounts
      FROM sales_audit.orders_daily
      WHERE order_date = CURRENT_DATE - 1

  # Any false condition fails the execution before publication.
  - id: enforce_contract
    type: io.kestra.plugin.core.execution.Assert
    errorMessage: "orders_daily breaks its contract, publication skipped"
    conditions:
      - "{{ outputs.contract_metrics.row.row_count > 0 }}"
      - "{{ outputs.contract_metrics.row.duplicate_ids == 0 }}"
      - "{{ outputs.contract_metrics.row.negative_amounts == 0 }}"

  # Publish: replace the published rows in one transaction, so readers
  # see either yesterday's version or today's, never a partial load.
  # sales.orders_daily is created once by the platform's provisioning step.
  - id: promote
    type: io.kestra.plugin.jdbc.postgresql.Queries
    url: "jdbc:postgresql://{{ secret('WAREHOUSE_HOST') }}:5432/analytics"
    username: "{{ secret('WAREHOUSE_USER') }}"
    password: "{{ secret('WAREHOUSE_PASSWORD') }}"
    transaction: true
    sql: |
      TRUNCATE sales.orders_daily;
      INSERT INTO sales.orders_daily SELECT * FROM sales_audit.orders_daily;

  # Platform-owned step shared by every domain.
  - id: publish
    type: io.kestra.plugin.core.flow.Subflow
    namespace: acme.platform
    flowId: publish_data_product
    inputs:
      product: orders_daily
      domain: sales
      row_count: "{{ outputs.contract_metrics.row.row_count }}"
    wait: true
    transmitFailed: true

outputs:
  - id: row_count
    type: INT
    value: "{{ outputs.contract_metrics.row.row_count }}"

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "30 5 * * *"

A few design choices carry the data mesh principles. The labels make ownership queryable: filtering executions by owner: sales-data returns every run of every product the team owns. The sla block turns the freshness promise into an execution state, and the sla: miss label lets a platform-level alerting flow catch breaches across all domains with one Flow trigger that matches on labels. The Assert task is the contract gate; teams that already write Soda Core 4 contracts can replace the SQL query and the assertion with the io.kestra.plugin.soda.VerifyContract task, which verifies a contract YAML against the data source and fails on check failures. The repeated warehouse connection block is what an Enterprise Edition reference policy removes: the platform team defines the connection once and flows opt in with policyRefs.

The platform team owns the publication subflow. Every domain calls the same steps, so freshness tracking and announcements stay consistent across the mesh:

id: publish_data_product
namespace: acme.platform

inputs:
  - id: product
    type: STRING
  - id: domain
    type: STRING
  - id: row_count
    type: INT

tasks:
  # Central freshness registry, readable by any consumer or dashboard.
  - id: record_freshness
    type: io.kestra.plugin.core.kv.Set
    key: "{{ inputs.domain }}_{{ inputs.product }}_published_at"
    value: "{{ execution.startDate }}"
    kvType: DATETIME

  - id: announce
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    url: "{{ secret('DATA_PRODUCTS_SLACK_WEBHOOK') }}"
    messageText: "{{ inputs.domain }}.{{ inputs.product }} published with {{ inputs.row_count }} rows (execution {{ execution.id }})"

The platform team can extend this subflow without touching any domain flow, for example by adding the dbt artifacts to DataHub blueprint as a step so catalog lineage refreshes on every publication.

On the consumer side, the finance domain builds revenue_daily from two products owned by two other domains. It does not schedule against their timetables. It waits for both to publish successfully before a deadline:

id: revenue_daily
namespace: acme.finance.revenue

labels:
  data_product: revenue_daily
  domain: finance
  owner: finance-data

tasks:
  - id: refresh_revenue
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://{{ secret('WAREHOUSE_HOST') }}:5432/analytics"
    username: "{{ secret('WAREHOUSE_USER') }}"
    password: "{{ secret('WAREHOUSE_PASSWORD') }}"
    sql: REFRESH MATERIALIZED VIEW finance.revenue_daily

triggers:
  # Runs only when both upstream products succeed before 07:00 UTC.
  - id: upstream_products_published
    type: io.kestra.plugin.core.trigger.Flow
    mode: ALL
    window:
      deadline: "07:00:00"
      timezone: UTC
    dependsOn:
      - namespace: acme.sales.orders
        flowId: orders_daily
        states: [SUCCESS]
      - namespace: acme.payments.refunds
        flowId: refunds_daily
        states: [SUCCESS]

If sales publishes but refunds fails its contract, finance does not build a revenue figure on half the inputs. That property is hard to get from a shared cron schedule and comes for free from event-based dependencies between products.

Federated computational governance, written as policy

Federated computational governance becomes real when the global rules are code that the platform evaluates on every change. The governance group reviews the rules in a pull request (the federated part), and the platform rejects non-compliant flows at save time with a clear error (the computational part).

With Kestra Enterprise Edition, the baseline for every data product can be a tenant-scoped policy that targets the domain namespaces:

id: data-product-baseline
displayName: "Data product baseline"
description: "Minimum guarantees for every data product flow."
enforcement: ACTIVE
target:
  namespaces:
    - acme.sales
    - acme.payments
    - acme.finance
rules:
  - type: io.kestra.plugin.ee.rules.Require
    on: FLOW
    properties:
      - labels.data_product
      - labels.owner
      - sla
    errorMessage: "Data product flows must declare data_product and owner labels and an SLA."
  - type: io.kestra.plugin.ee.rules.Restrict
    on: PLUGIN
    where:
      - field: type
        operator: EQUAL_TO
        value: io.kestra.plugin.dbt.cli.DbtCLI
    property: containerImage
    regex: "^ghcr\\.io/kestra-io/dbt-.*"
    errorMessage: "dbt must run on a platform-approved image."

target.namespaces matches each listed namespace and all its children, so onboarding a new domain is a one-line change reviewed like any other code. Domains keep their autonomy inside the guardrails: a domain can add stricter rules on its own namespace but cannot relax the baseline, because child namespaces only tighten inherited policies. A good rollout starts new rules in EVALUATE mode, which reports violations without blocking, and switches to ACTIVE once existing flows comply.

Open-source deployments do not include Policies. The same rules can run in CI instead: validate flow YAML with kestra flow validate and add a lint step that checks for required labels and the sla block before deployment.

Data mesh vs data fabric vs data lakehouse

Data mesh is an operating model that decides who owns data; data fabric is an integration architecture that decides how data is connected and accessed; a data lake or lakehouse is a storage and compute architecture that decides where data lives. They answer different questions, so one organization can use all three at once.

Data fabric, a term popularized by Gartner, describes a metadata-driven layer that connects distributed sources across clouds and on-premises systems and provides unified discovery, access, and governance, often through virtualization and automated metadata management. A central data or IT team usually runs the fabric, and its scope covers operational and analytical data alike.

Use the table as a decision aid, not as a menu of exclusive options:

• Choose data mesh when several business domains produce analytical data, a central team has become the bottleneck, and domains can staff pipeline ownership and on-call.
• Choose data fabric when the main pain is finding and accessing data scattered across many systems, and ownership can stay with a central team.
• Start with a lakehouse or warehouse alone when one team can still serve demand and the problem is consolidating storage, cost, or performance.
• Combine them when domain ownership and cross-system access both matter: a mesh where every domain publishes into one lakehouse, with a fabric-style catalog and access layer on top. The mesh defines who owns what and why, the fabric provides how data is found and reached, and the lakehouse is where it is stored.

Where data mesh adoptions stall

Data mesh programs rarely fail on technology. They stall when the organization adopts the vocabulary without the ownership change, or builds the pieces in the wrong order.

Renaming the central team without moving ownership is the most common failure. Domains get names and labels, but the former central team still writes and fixes every pipeline. That is the original bottleneck under a new org chart, plus the coordination overhead of the mesh. Much of the “data mesh is dead” commentary describes this pattern rather than the model itself.

Building the platform after the domains forces every domain to assemble its own stack. Six domains then produce six incompatible ways to schedule, test, and publish, and interoperability becomes a migration project. The minimum paved road (a namespace per domain, a build-and-check template, a publication step, catalog registration) has to exist before the second domain onboards.

Governance as a committee recreates the queue that data mesh was supposed to remove. If every new product needs a review meeting, domains wait for the meeting. Rules that can be checked automatically belong in policies and CI; the governance group’s job is to write and version those rules.

Data products without contracts leave consumers to discover breaking changes in their dashboards. A contract does not need to be elaborate to be useful: a schema, three or four quality guarantees, an owner, and a freshness SLA cover most incidents.

Coupling domains through schedules produces silent staleness. Finance scheduled at 07

Counting products instead of consumers rewards publishing tables nobody reads. Track which products have active consumers and which consumers depend on each product, and retire the rest.

Ignoring the cost model surprises leadership later. Domain ownership duplicates some compute and some headcount before the reduced central backlog pays it back, and the business case should say so upfront.

When not to build a data mesh

Skip data mesh when a single central data team still keeps up with demand, when the business is not organized into stable domains with engineering capacity, or when the real problem is integration of scattered systems (a fabric question) or storage consolidation (a lakehouse question). The mesh solves an ownership bottleneck; it adds cost everywhere else.

Signals that a mesh is premature:

• One data team of a few engineers serves the whole company, and its backlog is manageable.
• Domain teams have no engineers who can own pipelines, tests, and an on-call rotation.
• There is no budget for a platform team, so each domain would build its own tooling.
• Most incidents come from upstream source systems, which contracts at the source fix without reorganizing anyone.
• Reporting is centralized by regulation or by design, with one accountable owner for every number.

A partial adoption is often the better move. Data contracts, named product owners for the most-used datasets, and a namespace per team on a shared orchestrator deliver much of the reliability benefit without the full operating model. The trade-offs between centralized, decentralized, and hybrid team models are covered in more depth in data management organization structure.

A rollout path: one domain, one product, one consumer

The fastest way to test whether data mesh fits is to run it end to end on a narrow slice before reorganizing anyone.

1. Pick a domain with a motivated team and a real consumer in another domain who is waiting for its data.
2. Have the platform team ship the minimum paved road: a namespace for the domain, a build-and-check flow template, a publication subflow, and catalog registration.
3. Write the first product’s contract and SLA with the consumer, and commit them next to the transformation code and the flow.
4. Wire the consumer through an event-based trigger on the product’s successful publication, not through a schedule.
5. Turn the rules that emerged from steps 2 to 4 into policies or CI checks, then onboard the second domain.

Once a few domains publish this way, the supervision plane becomes a query over execution history: which products met their SLA, which contracts failed, and which consumers were blocked. To start from a working flow, adapt the dbt on Postgres blueprint into your first domain’s product, then add the contract gate and the publication subflow shown above.