Cloud Run Jobs: Serverless Batch Processing on Google Cloud

TL;DR — Cloud Run Jobs are a serverless execution environment on Google Cloud for running containerized batch, parallel, and run-to-completion tasks without managing underlying infrastructure. They are ideal for discrete, asynchronous workloads that don’t serve HTTP requests.

Managing batch processing at scale often involves a trade-off: either accept the operational overhead of virtual machines and Kubernetes clusters, or compromise on flexibility with highly constrained serverless functions. Google Cloud Run Jobs offer a compelling third path, providing a serverless environment to run containerized, run-to-completion tasks without managing the underlying infrastructure.

However, even serverless jobs require orchestration. Coordinating Cloud Run Jobs with other services, handling dependencies, managing retries, and providing end-to-end observability can quickly become complex. This article explores how Kestra’s declarative platform elevates Cloud Run Jobs from isolated tasks to integrated, production-ready workflows.

How Cloud Run Jobs simplify serverless batch processing

Cloud Run Jobs are a core component of Google Cloud’s serverless offering, designed specifically for executing containerized applications that run for a finite period and then exit. Unlike traditional server-based batch processing, which requires provisioning, configuring, and managing servers, Cloud Run Jobs abstract away all infrastructure concerns. You provide a container image, and Google Cloud handles the rest—from provisioning resources to scaling and termination.

This model is ideal for workloads such as:

• Data processing and transformation (ETL/ELT)
• Report generation
• Machine learning inference
• Database migrations
• Any run-to-completion task that can be containerized.

Each job execution creates one or more independent container instances, or “tasks,” which can run in parallel. This makes it a powerful tool for high-throughput, parallelizable batch workloads.

Cloud Run Jobs vs. Cloud Run Services: Understanding the Difference

It’s crucial to distinguish Cloud Run Jobs from their sibling, Cloud Run Services. While both run containers in a serverless environment, they serve fundamentally different purposes.

In short, use a Service when you need to respond to real-time requests from users or other systems. Use a Job when you need to perform a discrete, asynchronous task that runs in the background.

Why batch jobs on Google Cloud need robust orchestration

While Cloud Run Jobs provide an excellent execution environment, they are not a complete orchestration solution. A production-grade workflow requires more than just the ability to run a container. Key orchestration capabilities needed include:

• Advanced Scheduling and Triggering: Beyond simple cron schedules, you need to trigger jobs based on external events, such as a new file arriving in a Google Cloud Storage (GCS) bucket or the completion of another process.
• Dependency Management: Real-world pipelines involve multiple steps. A job might depend on the successful completion of a data ingestion task and, in turn, trigger a data transformation job.
• Error Handling and Retries: What happens if a job fails? A robust orchestrator provides configurable retry policies with exponential backoff, dead-letter queues, and conditional error branching to ensure reliability.
• Parameterization and Dynamic Execution: Workflows often need to pass data between tasks, such as passing a filename from a GCS trigger to a Cloud Run Job.
• End-to-End Observability: You need a single pane of glass to monitor the entire workflow, not just individual job executions. This includes centralized logging, execution history, and alerting on failures.
• Cross-Service Integration: Batch jobs rarely live in isolation. They need to interact with other services like BigQuery, Dataproc, dbt Cloud, or even systems in a multi-cloud environment.

Without an orchestration layer, you are left to build this complex logic yourself using a combination of Cloud Functions, Eventarc, and custom scripts—a solution that is often brittle and difficult to maintain.

Orchestrate Cloud Run Jobs with Kestra: Event-Driven Container Execution

Kestra provides a declarative control plane to manage Cloud Run Jobs as part of larger, more complex workflows. Instead of writing imperative glue code, you define the entire pipeline in a simple YAML file. Kestra treats the Cloud Run Job as a task within a larger flow, handling all the scheduling, triggering, and state management.

A key feature is Kestra’s Google Cloud Run Task Runner, an Enterprise Edition feature that allows any Kestra task to execute as a transient Cloud Run Job. This provides serverless, containerized execution for any script or command without requiring a dedicated worker fleet.

The following example demonstrates an event-driven workflow. It triggers when a new CSV file is uploaded to GCS, uses the Cloud Run Task Runner to process that file with a Python script, and then triggers a dbt Cloud job to update downstream models.

id: gcs-to-cloudrun-to-dbt
namespace: io.kestra.gcp.examples

description: |
  This workflow automates a common data pipeline pattern:
  1. Triggers when a new CSV file is uploaded to a GCS bucket.
  2. Runs a Python script as a serverless Cloud Run Job to process the file.
  3. Triggers a dbt Cloud job to update downstream data models.

tasks:
  - id: python-processing
    type: io.kestra.plugin.scripts.python.Script
    description: "Processes the input CSV file using a Python script."
    containerImage: python:3.11-slim
    beforeCommands:
      - pip install pandas
    script: |
      import pandas as pd
      df = pd.read_csv("{{ inputs.dataFile }}")
      # Example processing: add a new column
      df['processed_at'] = pd.Timestamp.now()
      df.to_csv("processed_data.csv", index=False)
    inputFiles:
      dataFile: "{{ trigger.uri }}"
    outputFiles:
      - processed_data.csv
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.CloudRun
      projectId: "{{ secret('GCP_PROJECT_ID') }}"
      region: "us-central1"
      serviceAccount: "{{ secret('GCP_SA_EMAIL') }}"

  - id: trigger-dbt-cloud
    type: io.kestra.plugin.dbt.cloud.TriggerRun
    description: "Triggers a dbt Cloud job to update downstream models."
    accountId: "{{ secret('DBT_ACCOUNT_ID') }}"
    jobId: "{{ secret('DBT_JOB_ID') }}"
    token: "{{ secret('DBT_API_TOKEN') }}"
    wait: true # Wait for the dbt job to complete

triggers:
  - id: watch-for-new-csv
    type: io.kestra.plugin.gcp.gcs.Trigger
    bucket: "your-landing-zone-bucket"
    prefix: "incoming/"
    interval: "PT1M"

A few things are worth noticing in this flow:

• Declarative & Event-Driven: The entire logic is defined in one YAML file. The gcs.Trigger automatically polls the bucket and starts a new flow execution for each new file.
• Serverless Task Execution: The Python script runs inside a container as a Cloud Run Job, managed by Kestra’s CloudRun task runner. Kestra handles the creation, execution, and cleanup of the job, providing a truly serverless experience.
• Seamless Integration: The workflow seamlessly integrates GCS, a custom Python script, and dbt Cloud. Kestra acts as the universal orchestrator connecting these disparate services.
• Data Context Passing: The GCS trigger automatically passes the URI of the new file ({{ trigger.uri }}) to the Python script task, maintaining data context throughout the flow.

Choosing the Right Task Runner for Google Cloud: Cloud Run vs. Batch

Kestra offers multiple Task Runner Types for executing workloads on Google Cloud, primarily the Google Cloud Run Task Runner and the Google Batch Task Runner.

• Use the Cloud Run Task Runner for:
  • Short-to-medium duration tasks (up to 24 hours).
  • Workloads that benefit from rapid startup times.
  • General-purpose containerized scripts and applications.
• Use the Google Batch Task Runner for:
  • Long-running, compute-intensive workloads.
  • High-performance computing (HPC), AI/ML training, and large-scale data processing.
  • Tasks requiring specific VM configurations, such as GPUs or large memory allocations.

For most general-purpose batch processing and data pipeline tasks, the Cloud Run Task Runner offers a cost-effective and low-overhead solution.

Where Cloud Run Jobs accelerate data and AI workflows

The combination of Cloud Run Jobs and a robust orchestrator like Kestra unlocks numerous use cases across data, AI, and infrastructure domains:

• ETL/ELT Data Transformations: Run containerized data transformation scripts (Python, R, etc.) on a schedule or triggered by new data arrivals.
• Scheduled Report Generation: Periodically run a job to query a database like BigQuery, generate a report, and upload it to GCS or send it via email.
• ML Inference Batch Predictions: Load a trained model into a container and run a job to generate predictions on a new batch of data.
• Backend Asynchronous Processing: Offload long-running tasks from a web service, such as video transcoding or image processing, to a Cloud Run Job.
• Modernizing Cron Jobs: Replace fragile cron jobs running on a single VM with reliable, observable, and scalable serverless containers.

Related concepts for modern GCP orchestration

Building production-grade systems on Google Cloud involves orchestrating a variety of services. Understanding how Cloud Run Jobs fit within this ecosystem is key.

• Google Cloud Storage (GCS): Often the starting point for data pipelines, with GCS events triggering Cloud Run Jobs.
• dbt Cloud Orchestration: Coordinate dbt Cloud jobs with upstream data loading processes handled by Cloud Run.
• Google Cloud Scheduler: While a basic scheduling tool, it highlights the need for more advanced, workflow-aware scheduling provided by an orchestrator.
• Deploying on GKE: For teams managing their own infrastructure, Kestra can be deployed on GKE to orchestrate both on-cluster and serverless workloads like Cloud Run Jobs.

Ready to build your first orchestrated Cloud Run Job? Explore Kestra’s blueprints to get started quickly.