Delta Lake: ACID Transactions, Lakehouse Architecture, and Orchestration

TL;DR — Delta Lake is an open-source storage layer that brings ACID transactions, schema enforcement, and time travel to data lakes, forming the foundation of modern lakehouse architectures for reliable data management.

Traditional data lakes promised flexibility but often struggled with data reliability, consistency, and governance, leading to “data swamps.” As data volumes and complexity grew, teams faced challenges ensuring data quality and managing concurrent operations without corruption.

Delta Lake emerged to solve these problems by bringing data warehousing features like ACID transactions directly to data lakes. This open-source storage layer transforms a simple collection of files into a robust, transactional data platform, forming the foundation of the modern lakehouse architecture. This article explores Delta Lake’s core capabilities and demonstrates how Kestra orchestrates its workflows for resilient data management.

How Delta Lake works: ACID transactions and unified data management

Delta Lake is an open-source storage layer that operates on top of existing data lakes, such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage. It doesn’t replace your object storage; instead, it enhances it with reliability and performance features. At its core, Delta Lake organizes data into versioned Parquet files and maintains a transaction log that records every operation performed on a table.

The transaction log: Enabling ACID properties

The key to Delta Lake’s reliability is its transaction log, stored as ordered JSON files in a _delta_log directory alongside the data files. This log is the single source of truth, providing a complete history of every change made to a table. It enables the ACID properties that are standard in traditional databases:

• Atomicity: Operations are all-or-nothing. A transaction that writes five files either succeeds completely or fails, leaving the table in its previous state.
• Consistency: Data is always in a valid state. Schema enforcement ensures that all data written to a table conforms to its defined structure.
• Isolation: Concurrent reads and writes do not interfere with each other. Readers see a consistent snapshot of the data, even while new data is being written.
• Durability: Once a transaction is committed, it is permanent, even in the event of system failures.

Schema enforcement and evolution

Delta Lake enforces the table’s schema on write, preventing the ingestion of corrupt or mismatched data that could break downstream processes. If an incoming record doesn’t match the expected data types or column names, the transaction fails. This proactive data quality check is a significant improvement over schema-on-read systems.

At the same time, Delta Lake supports explicit schema evolution. You can add new columns, change data types, or modify column comments without rewriting the entire table, allowing your data structures to evolve with business needs.

Time travel and versioning

Because the transaction log records every version of the table, Delta Lake allows you to query historical data. This “time travel” capability is invaluable for:

• Auditing: Examining data at a specific point in time for compliance or analysis.
• Rollbacks: Quickly reverting a table to a previous version to recover from accidental bad writes or deletions.
• Reproducibility: Running machine learning models or reports on the exact same version of the data for consistent results.

These features collectively build a reliable foundation for a modern Lakehouse Architecture.

Why Delta Lake needs orchestration beyond basic operations

While Delta Lake provides a robust foundation, managing data pipelines in a production environment requires more than just its core features. This is where orchestration becomes critical. Production workflows involve complex dependencies, error handling, and operational governance that extend beyond the capabilities of a storage layer.

An orchestration platform is needed to:

• Manage complex dependencies: A typical ETL/ELT pipeline involves multiple steps, from ingesting raw data and running transformations to loading it into Delta tables and performing quality checks. Orchestration ensures these steps run in the correct order.
• Handle error recovery: If a write to a Delta table fails, an orchestrator can trigger automated recovery logic, retry the operation, or send an alert to the data team.
• Automate data quality checks: Orchestration platforms can integrate with tools like dbt or Great Expectations to run validation tests before and after data is written to Delta Lake.
• Integrate diverse systems: Data rarely originates in one place. Orchestration connects various sources (databases, APIs, streaming platforms) and sinks, with Delta Lake as a central component.
• Provide end-to-end visibility: A centralized control plane gives you a complete view of your data lifecycle, making it easier to monitor performance, debug issues, and manage compliance across all your data pipelines.

Without a dedicated orchestrator, teams are left to build and maintain brittle, custom scripts, losing the reliability that Delta Lake was adopted to provide. For more insight, you can explore various ETL orchestration tools.

Orchestrate Delta Lake with Kestra: An ETL example

Kestra can manage the entire lifecycle of a Delta Lake workflow, from data ingestion and processing to loading and validation. The following example demonstrates a common ETL pattern: downloading raw data from S3, processing it with Python, and loading it into a Delta Lake table hosted on Databricks.

id: delta_lake_etl_workflow
namespace: company.team.data
description: "Ingests data from S3, processes it with Python, and loads into a Delta Lake table on Databricks."

inputs:
  - id: s3_bucket
    type: STRING
    defaults: "my-data-bucket"
  - id: s3_key
    type: STRING
    defaults: "raw_data/sales_data.csv"
  - id: databricks_table
    type: STRING
    defaults: "sales_data.processed_sales"

tasks:
  - id: download_from_s3
    type: io.kestra.plugin.aws.s3.Download
    bucket: "{{ inputs.s3_bucket }}"
    key: "{{ inputs.s3_key }}"

  - id: process_data_with_python
    type: io.kestra.plugin.scripts.python.Script
    warningOnStdErr: false
    script: |
      import pandas as pd
      import io
      from kestra.io import KestraIO

      # Read raw data from Kestra's internal storage
      with open("{{ outputs.download_from_s3.uri }}", "r") as f:
          raw_csv_data = f.read()

      df = pd.read_csv(io.StringIO(raw_csv_data))
      df['processed_at'] = pd.to_datetime('now')
      df['total_amount'] = df['quantity'] * df['price']

      # Write processed data to a temporary file
      processed_file_path = "processed_sales.csv"
      df.to_csv(processed_file_path, index=False)

      # Make the processed file available as an output
      KestraIO.outputs({"uri": processed_file_path})
    outputs:
      - processed_sales.csv

  - id: load_to_delta_lake_databricks
    type: io.kestra.plugin.jdbc.databricks.Query
    url: "jdbc:databricks://{{ secret('DATABRICKS_HOST') }}:443/default;AuthMech=3;UID=token;PWD={{ secret('DATABRICKS_TOKEN') }};"
    sql: |
      CREATE TABLE IF NOT EXISTS {{ inputs.databricks_table }} (
        id INT,
        product_name STRING,
        quantity INT,
        price DOUBLE,
        processed_at TIMESTAMP,
        total_amount DOUBLE
      ) USING DELTA;

      COPY INTO {{ inputs.databricks_table }}
      FROM '{{ outputs.process_data_with_python.uri }}'
      FILEFORMAT = CSV
      FORMAT_OPTIONS ('header' = 'true', 'inferSchema' = 'false')
      COPY_OPTIONS ('mergeSchema' = 'true');
    beforeCommands:
      - "SET spark.databricks.delta.schema.autoMerge.enabled = true;"

errors:
  - id: send_alert_on_failure
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "text": ":x: Delta Lake ETL workflow failed! \n Flow: `{{ flow.id }}` \n Execution: `{{ execution.id }}` \n Details: {{ execution.status }}"
      }

A few things are worth noticing in this workflow:

• Declarative & Version Controlled: The entire ETL process is defined in a single YAML file, making it easy to version, review, and audit.
• Polyglot Execution: The flow seamlessly combines different tools—an S3 integration, a Python script for transformation, and a Databricks SQL query—without requiring complex glue code.
• Seamless Data Handoff: Kestra’s internal storage automatically passes the output of one task as an input to the next, simplifying data flow between steps.
• Built-in Error Handling: The errors block provides a robust mechanism for sending alerts or triggering remediation workflows when any task fails.

This declarative approach to orchestration complements Delta Lake’s reliability, creating a truly robust and manageable data platform. For more details on our partnership, see how Databricks and Kestra are joining forces.

Delta Lake in action: Common use cases and integrations

Delta Lake’s features make it suitable for a wide range of data workloads, from traditional analytics to real-time applications.

Building a Lakehouse architecture

This is the primary use case. Delta Lake provides the transactional layer needed to perform both BI and AI workloads directly on the data lake, eliminating the need for separate, siloed data warehouses and data lakes.

Handling real-time data ingestion

With its support for streaming reads and writes, Delta Lake is well-suited for building real-time data pipelines. It can serve as a unified sink for both batch and streaming data, simplifying the architecture for applications that require fresh data.

Data versioning and audit trails

For regulated industries, the ability to reconstruct the state of data at any point in time is critical for compliance and auditing. Time travel provides an immutable history of all data changes.

Integrations across the data ecosystem

Delta Lake is an open-source project with a growing ecosystem. It integrates natively with compute engines like Apache Spark, Trino, and Flink. It is also a core component of platforms like Databricks and is supported by Dremio, and can be managed with tools like dbt. You can learn more about data lakehouse orchestration with Kestra and Dremio or explore dbt integrations for data orchestration. The format is also a key part of modern data platforms, including as an alternative to Microsoft Fabric.

Delta Lake vs. other table formats: Iceberg and Hudi

Delta Lake is one of three major open table formats for the data lakehouse. Its main counterparts are Apache Iceberg and Apache Hudi.

• Apache Iceberg: Originally developed at Netflix, Iceberg is known for its strong focus on correctness and performance, especially for huge tables. It is compute-engine agnostic and has broad support across the ecosystem.
• Apache Hudi: Created at Uber, Hudi (Hadoop Upserts Deletes and Incrementals) focuses on providing fast upserts and incremental data processing, making it a strong choice for change data capture and streaming use cases.

The choice between them often depends on your primary compute engine and specific needs. Delta Lake has the tightest integration with Spark and the Databricks ecosystem, while Iceberg is often favored for its interoperability. Many platforms are now offering support for all three, as seen in various Databricks alternatives.

Kestra’s role in modern Delta Lake environments

As organizations adopt Delta Lake to build reliable data platforms, Kestra provides the essential orchestration layer to operationalize these environments at scale. By offering a unified, declarative control plane, Kestra helps teams manage the entire data lifecycle around Delta Lake. This includes centralizing data ingestion, automating transformations, enforcing data quality, and providing end-to-end visibility and governance.

By combining the reliability of Delta Lake with the robust orchestration capabilities of Kestra, data teams can build modern, scalable, and resilient data platforms to power their analytics and AI initiatives. Explore how Kestra can help you build your modern data platform or browse our full set of data engineering resources.

Related concepts

• Lakehouse Architecture: Principles & Benefits
• Data Orchestration: The Engine of Modern Data Pipelines
• Databricks Alternatives: Top Data & AI Platforms (2026)
• dbt Integrations for Data Orchestration
• Top ETL Orchestration Tools for Modern Data Pipelines
• Orchestrate Data Pipelines in Kestra