What Is Apache Kafka? Architecture, Concepts & Orchestration

TL;DR — Apache Kafka is an open-source distributed event streaming platform for real-time data pipelines and applications, designed for high-throughput, fault-tolerant ingestion, storage, and processing of event streams.

In today’s event-driven architectures, the ability to handle streams of data in real time is paramount. From user activity logs to IoT sensor readings, organizations generate a continuous flow of information that needs to be ingested, processed, and acted upon without delay. This constant deluge of data presents both immense opportunities and significant challenges for traditional batch processing systems. Apache Kafka emerged as a solution to this problem, designed from the ground up to be a distributed, fault-tolerant, and high-throughput platform for handling event streams.

How Apache Kafka Works: Producers, Brokers, and Consumers

Kafka’s power lies in its simple yet robust architecture. It functions as a distributed, append-only log, allowing multiple applications to publish (write) and subscribe to (read) streams of records in a scalable and reliable manner.

Core Components of Kafka

• Producers: Applications that publish streams of records to one or more Kafka topics.
• Consumers: Applications that subscribe to topics and process the stream of records published to them. Consumers read records in the order they were produced.
• Brokers: The servers that make up a Kafka cluster. Each broker stores data and serves client requests. Brokers manage data replication and partition distribution.
• Topics: A category or feed name to which records are published. Topics in Kafka are multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.
• Partitions: Each topic is split into one or more partitions. Partitions allow for parallel processing by splitting the data for a topic across multiple brokers. This is a key mechanism for Kafka’s scalability.
• Offset: A unique, sequential ID given to each record within a partition. Consumers track their position in a partition using this offset, allowing them to read messages from a specific point.
• ZooKeeper/Kraft: Historically, ZooKeeper was used for managing and coordinating Kafka brokers. More recent versions can use Kafka’s own quorum controller, Kraft, to remove the ZooKeeper dependency, simplifying the deployment architecture.

The Distributed Architecture

Kafka achieves high availability and durability through its distributed nature. A Kafka cluster consists of one or more brokers. Topics are partitioned, and these partitions are distributed across the brokers in the cluster. For fault tolerance, each partition can be replicated across multiple brokers.

One broker acts as the “leader” for a given partition, handling all read and write requests. Other brokers act as “followers,” passively replicating the leader’s data. If the leader fails, one of the followers is automatically promoted to become the new leader, ensuring continuous availability. This model provides strong guarantees for message durability and fault tolerance, making Kafka a reliable choice for critical event-driven orchestration.

Why Streaming Data Needs Orchestration

While Kafka provides a powerful foundation for streaming data, building production-grade pipelines requires more than just a message bus. Effective orchestration is crucial to manage the complexity and ensure reliability.

• End-to-End Integrity: Orchestration ensures that messages are not just consumed but are fully processed, transformed, and loaded into downstream systems with guaranteed semantics (e.g., exactly-once processing).
• Complex Dependencies: Real-world pipelines involve multiple steps. An orchestrator manages the dependencies between Kafka consumers, data transformation jobs, database writes, and API calls.
• Robust Error Handling: What happens when a consumer fails? An orchestration platform implements sophisticated retry mechanisms, such as exponential backoff, and routes failed messages to a dead-letter queue for analysis without halting the entire pipeline.
• Monitoring and Alerting: Orchestration provides centralized visibility into the health of your streaming pipelines. It automates monitoring, sends alerts on failures, and can trigger operational runbooks.
• Data Quality and Governance: An orchestrator can enforce data quality checks on incoming streams and manage schema evolution, ensuring that your data observability is maintained as your systems evolve.

Orchestrate Kafka with Kestra: A Real-time Data Pipeline Scenario

Using Kafka effectively in a production environment involves coordinating consumers, producers, and the business logic that processes the data. Kestra allows you to define and manage this entire workflow declaratively in YAML.

Consider a scenario where you consume user activity events from a Kafka topic, filter out low-value events using a Python script, and then publish the enriched events to a separate topic for downstream analytics.

id: kafka-event-processing
namespace: company.team.analytics

tasks:
  - id: consume_events
    type: io.kestra.plugin.kafka.Consume
    properties:
      bootstrap.servers: "{{ secret('KAFKA_BOOTSTRAP_SERVERS') }}"
      security.protocol: "SASL_SSL"
      sasl.mechanism: "PLAIN"
      sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username='{{ secret('KAFKA_USERNAME') }}' password='{{ secret('KAFKA_PASSWORD') }}';"
    topic: user_activity_raw
    maxRecords: 500
    keyDeserializer: STRING
    valueDeserializer: JSON

  - id: filter_and_enrich
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      data.json: "{{ outputs.consume_events.uri }}"
    script: |
      import json
      import sys

      with open('data.json', 'r') as f:
        records = [json.loads(line) for line in f]

      processed_records = []
      for record in records:
        # Example: filter for 'purchase' events and add a timestamp
        if record.get('event_type') == 'purchase':
          record['processed_at'] = "{{ now() | date('iso') }}"
          processed_records.append(record)

      # Write processed records to a new file for the next task
      with open('processed_data.json', 'w') as f:
        for record in processed_records:
          f.write(json.dumps(record) + '\n')

  - id: produce_enriched_events
    type: io.kestra.plugin.kafka.Produce
    properties:
      bootstrap.servers: "{{ secret('KAFKA_BOOTSTRAP_SERVERS') }}"
      security.protocol: "SASL_SSL"
      sasl.mechanism: "PLAIN"
      sasl.jaas.config: "org.apache.kafka.common.security.plain.PlainLoginModule required username='{{ secret('KAFKA_USERNAME') }}' password='{{ secret('KAFKA_PASSWORD') }}';"
    topic: user_activity_enriched
    from: "{{ outputs.filter_and_enrich.outputFiles['processed_data.json'] }}"
    keySerializer: STRING
    valueSerializer: JSON

A few things are worth noticing in this flow:

• Declarative Definition: The entire pipeline, from consuming to producing, is defined in a single, version-controllable YAML file.
• Polyglot Processing: A Kafka consumer task, written in Java, seamlessly passes data to a Python script for custom business logic.
• State Management: Kestra handles passing the data between tasks via internal storage, abstracting away the complexity of file management.
• Secrets Management: Credentials are not hardcoded; they are securely accessed using Kestra’s secret management system.
• Built-in Error Handling: If any task fails, Kestra’s built-in retry and error handling mechanisms can be configured to manage the failure gracefully.

This declarative approach simplifies Kafka orchestration, making complex streaming pipelines easier to build, maintain, and scale.

Where Kafka Stream Orchestration Pays Off

Integrating Kafka with a robust orchestration platform unlocks numerous high-value use cases across an organization:

• Real-time Analytics: Feed cleaned and transformed data from Kafka streams directly into real-time dashboards and analytical systems.
• Event Sourcing: Use Kafka as the system of record for events in a microservices architecture, with an orchestrator managing the complex interactions between services.
• Log Aggregation: Ingest logs from thousands of services, process them in real-time to detect anomalies, and forward them to long-term storage or monitoring systems.
• Change Data Capture (CDC): Stream database changes into Kafka and orchestrate the process of replicating those changes to data warehouses, search indexes, or other systems. See how to build CDC pipelines.
• IoT Data Ingestion: Handle massive streams of data from IoT devices, orchestrating filtering, aggregation, and alerting based on real-time sensor readings.

Related Concepts

• Kafka Connect: Data Streaming & Orchestration
• Kafka Streams: Real-Time Processing Explained
• Kinesis vs. Kafka: Streaming Platform Comparison
• Pub/Sub vs. Kafka: Event Streaming Platform Comparison
• What is Data Ingestion?
• What Is Data Orchestration?