Python Logging: Centralized Management for Data & AI Workflows

TL;DR — Python logging provides a flexible, built-in framework to record events from applications. For production data and AI workflows, orchestrating Python logging with a platform like Kestra centralizes collection, standardizes output, and enables advanced management and observability across distributed systems.

Python applications, especially in data and AI, generate a continuous stream of information. Without proper logging, this stream becomes a black box, making debugging, monitoring, and auditing nearly impossible. Relying solely on print statements or basic file writes quickly breaks down in distributed, production-grade environments.

This guide explores Python’s built-in logging module, its core components, and how to configure it effectively. More importantly, we’ll demonstrate how to elevate your Python logging by orchestrating it with Kestra, ensuring logs are not just generated, but are consistently captured, centralized, and actionable across your entire workflow ecosystem.

How Python Logging Works: Core Concepts and Mechanisms

Python’s standard library includes a powerful logging module that provides a flexible framework for emitting log messages from applications and libraries. Understanding its core components is the first step toward effective log management.

Loggers, Handlers, and Formatters Explained

The logging module’s architecture is built on three key components:

• Loggers: These are the primary entry points into the logging system. When your code needs to log an event, it requests a logger by name and calls a method on it (e.g., logger.info()). Loggers are organized in a hierarchy using dot notation (e.g., my_app.database), allowing for fine-grained configuration.
• Handlers: Handlers are responsible for dispatching log records to the appropriate destination. A logger can have multiple handlers. Common built-in handlers include StreamHandler (sends logs to the console), FileHandler (writes logs to a file), and SysLogHandler (sends logs to a syslog daemon).
• Formatters: Formatters define the final structure and content of the log message. You can specify a format string that includes attributes like the timestamp (asctime), log level (levelname), module name (module), and the log message itself (message).

These components work together: a logger receives a message, passes it to its handlers, and each handler uses its formatter to render the message before sending it to its destination. This modular design allows you to run Python scripts with logging configurations that can be changed independently of the application code. Kestra’s Script task for Python leverages this by capturing standard output streams, making it easy to integrate existing Python code. You can learn more about how to send logs from Python scripts to Kestra in our documentation.

Understanding Logging Levels and Filters

To manage the volume and relevance of log output, the logging module uses a system of severity levels. Each log message is assigned a level, and you can configure loggers and handlers to only process messages above a certain severity. The standard levels, in increasing order of severity, are:

• DEBUG: Detailed information, typically of interest only when diagnosing problems.

• INFO: Confirmation that things are working as expected.

• WARNING: An indication that something unexpected happened, or a potential problem in the near future (e.g., ‘disk space low’). The software is still working as expected.

• ERROR: Due to a more serious problem, the software has not been able to perform some function.

• CRITICAL: A very serious error, indicating that the program itself may be unable to continue running.

By default, the root logger is set to WARNING, meaning it will ignore DEBUG and INFO messages. This is a common source of confusion for beginners. For Kestra users, it’s important to know that script outputs to stderr are now logged at the ERROR level, a change introduced to better reflect the severity of such messages. You can read more about this in the Kestra 0.21.0 migration guide.

Why Effective Logging Needs Orchestration

While Python’s logging module is powerful, managing logs in a production environment introduces a new set of challenges that can’t be solved by the library alone. This is where orchestration becomes critical.

• Debugging Distributed Systems: Modern applications are often composed of multiple services. When a failure occurs, you need to trace a request across different components. Without a centralized system, this involves manually searching and correlating logs from multiple machines, which is time-consuming and error-prone.
• Centralized Observability: To get a complete picture of system health, logs must be collected from all sources and aggregated in a central location. An orchestration platform acts as this central hub, capturing logs from every task in a workflow. For enterprise-grade needs, Kestra can further ship these logs to external observability platforms like Datadog or Splunk.
• Auditing and Compliance: In many industries, maintaining an immutable and auditable record of operations is a strict requirement. An orchestrator provides a single source of truth, logging every action, its inputs, outputs, and status, providing a complete audit trail.
• Contextual Alerting: Raw logs are noisy. Effective alerting requires context. An orchestration platform knows when a log message is associated with a critical production workflow, allowing you to build alerts that trigger only on significant events, reducing alert fatigue.

Orchestrate Python Logging with Kestra: Centralized Collection and Management

Kestra simplifies Python logging by automatically capturing and centralizing all log output from your scripts. Any message written to stdout or stderr is ingested, timestamped, and associated with the specific task execution that generated it.

The following Kestra flow demonstrates how to run a Python script that generates logs at various levels. Kestra captures these logs and makes them visible in the UI, providing immediate context for debugging and monitoring.

id: python-logging-demonstration
namespace: company.team.data

tasks:
  - id: python_script_with_logging
    type: io.kestra.plugin.scripts.python.Script
    description: "Demonstrates capturing different log levels from a Python script."
    docker:
      image: python:3.11-slim
    script: |
      import logging
      import sys

      # Configure logging to output to stdout
      logging.basicConfig(
          level=logging.DEBUG,
          format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
          stream=sys.stdout
      )

      logger = logging.getLogger('DataProcessing')

      logger.debug("This is a detailed debug message.")
      logger.info("Starting data processing task.")
      logger.warning("Configuration value is low, but acceptable.")
      logger.error("Failed to connect to a non-critical API endpoint.")

      print("--- Standard print statement also captured as INFO level log ---")

When you run this flow, the “Logs” tab for the python_script_with_logging task will display all the messages, each with its corresponding log level and timestamp.

What’s worth noticing in this approach:

• Zero Configuration: Kestra automatically captures stdout and stderr without any special configuration in your Python script.
• Centralized View: Logs from this Python script are stored alongside logs from all other tasks in your workflow (e.g., SQL queries, shell commands), providing a unified view of the entire process.
• Execution Context: Each log entry is automatically tagged with metadata like the flow ID, execution ID, and task ID, making it easy to trace logs back to their source.
• Custom Logging: For more advanced use cases, you can use a library like Loguru to create structured, color-coded logs that are still captured seamlessly by Kestra.

Enhancing Python Logs with Kestra’s Native Capabilities

Beyond simply capturing logs, Kestra provides tools to create more structured and meaningful output from your Python scripts.

Kestra’s outputs system allows tasks to pass structured data and files to downstream tasks. This is a powerful form of “logging” results and metrics. Instead of parsing log files for specific values, you can have your Python script explicitly declare its results as outputs. This makes your workflows more robust and easier to maintain. Learn more about how to pass data between tasks.

Additionally, Kestra has a native Log task. This can be useful for adding structured log messages directly into the workflow execution from the orchestration layer, independent of any script. For example, you can use a Log task to record the start or end of a business process, using dynamic variables from other tasks.

Advanced Patterns for Python Logging in Kestra Workflows

Managing Dependencies and Environments for Consistent Logging

Consistent logging behavior requires a consistent environment. Kestra helps you manage this by allowing you to specify Python dependencies directly in your workflow definition. You can provide a requirements.txt file or use a tool like uv to ensure that the correct versions of your logging libraries and other packages are always installed. See our guides on managing Python dependencies and using uv for more details.

For complete isolation and reproducibility, you can run your Python tasks in Docker containers. By specifying a Docker image, you lock down the entire environment, including the operating system and system-level libraries, ensuring your logging configuration behaves identically everywhere.

Centralizing Logs for Observability and Alerting

While Kestra provides a centralized view of logs for each workflow execution, large organizations often need to aggregate logs from all systems into a dedicated observability platform. Kestra Enterprise supports this with the Log Shipper feature, which can forward all workflow and task logs to systems like Elasticsearch, Datadog, or Splunk.

This enables you to build dashboards and alerts based on log patterns across your entire infrastructure. You can correlate logs from your Kestra workflows with logs from your applications, databases, and infrastructure, providing a holistic view of system health and performance.

Related Concepts

• Run Python Inside Your Flows
• Kestra and Python: How to Run and Integrate Complex Scripts into Your Flows
• Data Observability
• Run a Python Script and Generate Outputs and Metrics
• Build your first Hello World Flow
• Workflow Orchestration Security