Run Python script in a Docker container based on Google Artifact Registry

Run a Python data analysis script inside a Docker container whose image lives in a private Google Artifact Registry repository, with authentication handled automatically by a short-lived OAuth token. This blueprint solves a common GCP pain point: how to let an orchestrator pull a custom container image from Artifact Registry without hardcoding long-lived Docker credentials. It downloads a CSV dataset, runs pandas analysis in your own image, and captures metrics back into Kestra.

How it works

1. The wdir task (io.kestra.plugin.core.flow.WorkingDirectory) groups the steps so files are shared in a single working directory.
2. download_csv (io.kestra.plugin.core.http.Download) fetches a public orders.csv dataset over HTTP.
3. fetch_auth_token (io.kestra.plugin.gcp.auth.OauthAccessToken) exchanges your GCP service account for a short-lived OAuth access token used to authenticate against Artifact Registry.
4. analyze_sales (io.kestra.plugin.scripts.python.Script) runs pandas on the CSV, computes total sales and median quantity, prints the top orders, and emits values with Kestra.outputs(...). It runs on the io.kestra.plugin.scripts.runner.docker.Docker task runner, whose config.auths injects the OAuth token as the Docker password so the private containerImage pulls cleanly.

What you get

• A reproducible Python run inside your own Artifact Registry image.
• Secure, token-based registry auth with no static Docker passwords.
• Captured outputs (total_sales, median_quantity) available to downstream tasks.
• Automatic input file mapping from the HTTP download into the container.

Who it's for

• Data engineers running Python pipelines on custom GCP images.
• Platform teams standardizing on Artifact Registry for container delivery.
• ML practitioners who need pinned dependencies in a private image.

Why orchestrate this with Kestra

Artifact Registry stores and serves images, but it cannot schedule work, retry failed runs, react to events, or track data lineage across tasks. Kestra fills that gap: declarative YAML defines the pipeline, event and schedule triggers can launch it, retries handle transient registry or network errors, and outputs flow between tasks with full execution lineage. The OAuth token is fetched fresh per run, so credentials stay short-lived and never sit in the image.

Prerequisites

• A GCP project with an Artifact Registry repository containing your Python image.
• A service account with the Artifact Registry Reader (or Writer to push) role.

Secrets

• GCP_CREDS: the JSON service account key, used by fetch_auth_token to mint the OAuth access token.

Quick start

1. Add the GCP_CREDS secret with your service account JSON key.
2. Set projectId, the containerImage path, and the registry region in config.auths.
3. Build and push your image: docker build -t yourGcpRegion-docker.pkg.dev/PROJECT/REPO/python:latest . then docker push ....
4. Execute the flow and inspect the captured outputs.

How to extend

• Swap the public CSV for a GCS object or database extract.
• Add a schedule or event trigger to run the analysis automatically.
• Push results to BigQuery or a warehouse in a follow-up task.
• Parameterize the image tag with an input to test multiple versions.

Links

• Google Cloud plugin documentation