Blueprints

Run Python script in a Docker container based on Google Artifact Registry container image

Source

yaml
id: python-docker-artifact-registry-gcp
namespace: company.team
tasks:
  - id: wdir
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: download_csv
        type: io.kestra.plugin.core.http.Download
        uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
      - id: fetch_auth_token
        type: io.kestra.plugin.gcp.auth.OauthAccessToken
        projectId: YOUR_GCP_PROJECT_NAME
        serviceAccount: "{{ secret('GCP_CREDS') }}"
      - id: analyze_sales
        type: io.kestra.plugin.scripts.python.Script
        inputFiles:
          data.csv: "{{ outputs.download_csv.uri }}"
        script: |
          import pandas as pd
          from kestra import Kestra

          df = pd.read_csv("data.csv")
          sales = df.total.sum()
          med = df.quantity.median()

          Kestra.outputs({"total_sales": sales, "median_quantity": med})

          top_sellers = df.sort_values(by="total", ascending=False).head(3)
          print(f"Top 3 orders: {top_sellers}")
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
          config: >
            {
              "auths": {
                "yourGcpRegion-docker.pkg.dev": {
                  "username": "oauth2accesstoken",
                  "password": "{{ outputs.fetch_auth_token.accessToken.tokenValue }}"
                }
              }
            }
        containerImage: yourGcpRegion-docker.pkg.dev/YOUR_GCP_PROJECT_NAME/REPO_NAME/python:latest

About this blueprint

Docker Python Software Engineering GCP

This flow downloads a CSV file from a public URL, analyzes it with a Python script, and outputs the results. The Python script is executed in a Docker container. The Docker image is stored in Google Artifact Registry.

To push an image to Google Artifact Registry, you need to:

  • Create a Google Cloud Platform service account with the Artifact Registry Writer role.
  • Create a JSON key for the service account.
  • Create a secret with the contents of the JSON key.
  • Build a Docker image: docker build -t yourGcpRegion-docker.pkg.dev/YOUR_GCP_PROJECT_NAME/REPO_NAME/python:latest .
  • Push the image to Google Artifact Registry: docker push yourGcpRegion-docker.pkg.dev/YOUR_GCP_PROJECT_NAME/REPO_NAME/python:latest

Note that the OauthAccessToken task is necessary to securely fetch a short-lived access token.

Working Directory

Download

Oauth Access Token

Script

Docker

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra