S3 Presigned URLs: Secure, Time-Limited Access to Your Objects

TL;DR — S3 Presigned URLs provide temporary, secure access to Amazon S3 objects, allowing users or applications to upload or download files without needing direct AWS credentials. They are generated with specific permissions and expiration times, simplifying secure data sharing and external integrations.

Sharing files securely or allowing external systems to upload data to Amazon S3 often creates a dilemma: how to grant access without exposing long-term AWS credentials? Direct access requires complex IAM policies and credential management, increasing your attack surface. For applications or users needing temporary, specific access to S3 objects, a more elegant solution is needed.

This is where S3 presigned URLs come in. They offer a powerful, time-limited mechanism to delegate access, ensuring that sensitive data remains protected while enabling seamless data exchange. This article will demystify presigned URLs, explain their mechanics, and demonstrate how Kestra can automate their generation and lifecycle for robust, secure data operations.

How S3 Presigned URLs Work

An S3 presigned URL is a URL that you can provide to users or applications to grant temporary access to a specific S3 object. The person who creates the presigned URL must have the necessary permissions to perform the operation that the URL enables.

The core components of a presigned URL are:

• Security Credentials: The URL is signed using the AWS credentials of an IAM user or role. The permissions of this user or role are temporarily conferred upon anyone who uses the URL.
• HTTP Method: When creating the URL, you specify an HTTP method, such as GET for downloading an object or PUT for uploading one. The URL is only valid for that specific action.
• Bucket and Object Key: The URL points to a specific object within a specific S3 bucket. It cannot be used to access other objects or list bucket contents.
• Expiration Time: You define a validity period, from a few seconds to a maximum of seven days (if using IAM instance profile credentials). After this time, the URL expires and no longer grants access.

When a client uses a presigned URL, Amazon S3 checks the signature and expiration time. If both are valid, S3 performs the action as if it were carried out by the original IAM user or role that generated the URL.

Why Secure Object Access Needs Orchestration

While you can generate presigned URLs manually with the AWS CLI or SDKs, managing them at scale in a production environment introduces operational challenges. This is where orchestration becomes critical.

• Dynamic Generation: Workflows often require presigned URLs to be created on-demand, triggered by an API call, a new message in a queue, or another system event. Orchestration platforms can listen for these events and generate URLs as needed.
• Lifecycle Management: Orchestration ensures that URLs are generated with the shortest possible, appropriate expiration time. It can also manage the entire lifecycle, such as triggering a data processing task immediately after a file is uploaded via a presigned URL.
• Integration with Broader Processes: Generating a URL is rarely the final step. It’s usually part of a larger process, like notifying a user via email with the download link or validating an uploaded file. An orchestrator connects these steps into a single, reliable workflow.
• Centralized Auditing and Logging: An orchestration platform provides a centralized place to log every URL generation event, creating an audit trail of who accessed what and when.
• Secure Credential Management: Instead of scattering AWS credentials across various scripts and applications, an orchestrator uses a centralized secrets manager, ensuring that the powerful credentials needed to generate URLs are never exposed.

Orchestrate S3 Presigned URL Generation with Kestra: Secure Upload and Download

Kestra simplifies the management of S3 presigned URLs by integrating their generation directly into your automated workflows. The io.kestra.plugin.aws.s3.utils.PresignedUrl task allows you to create URLs for both uploads and downloads with just a few lines of YAML.

The following Kestra flow demonstrates a practical scenario: it generates one presigned URL to allow an application to upload a daily report and another for a user to download a processed data artifact.

id: s3_presigned_url_automation
namespace: company.team

description: Generate and use S3 presigned URLs for secure upload and download operations.

inputs:
  - id: bucket
    type: STRING
    defaults: "your-kestra-bucket"
  - id: uploadKey
    type: STRING
    defaults: "uploads/daily-report-{{ now() | date('yyyy-MM-dd') }}.csv"
  - id: downloadKey
    type: STRING
    defaults: "artifacts/processed-data.json"
  - id: expirySeconds
    type: INT
    defaults: 3600 # 1 hour

tasks:
  - id: generate_upload_url
    type: io.kestra.plugin.aws.s3.utils.PresignedUrl
    aws:
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
      region: "eu-central-1"
    bucket: "{{ inputs.bucket }}"
    key: "{{ inputs.uploadKey }}"
    method: PUT
    expiresIn: "{{ inputs.expirySeconds }}"
    outputId: uploadUrlOutput

  - id: generate_download_url
    type: io.kestra.plugin.aws.s3.utils.PresignedUrl
    aws:
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
      region: "eu-central-1"
    bucket: "{{ inputs.bucket }}"
    key: "{{ inputs.downloadKey }}"
    method: GET
    expiresIn: "{{ inputs.expirySeconds }}"
    outputId: downloadUrlOutput

  - id: simulate_upload_and_download
    type: io.kestra.plugin.scripts.bash.Commands
    runner: DOCKER
    docker:
      image: curlimages/curl:latest
    commands:
      - |
        echo "Simulating upload to: {{ outputs.generate_upload_url.uri }}"
        # Create a dummy file to upload
        echo "id,value" > /tmp/report.csv
        echo "1,100" >> /tmp/report.csv
        curl -v -X PUT -H "Content-Type: text/csv" --upload-file /tmp/report.csv "{{ outputs.generate_upload_url.uri }}"
      - |
        echo "Simulating download from: {{ outputs.generate_download_url.uri }}"
        # This will likely fail unless the downloadKey object already exists in S3
        curl -v "{{ outputs.generate_download_url.uri }}" -o /tmp/downloaded-file.json

A few things are worth noticing in this flow:

• Kestra’s PresignedUrl task simplifies URL generation for both PUT (upload) and GET (download) methods, abstracting away the complexities of the AWS SDK.
• Secrets are securely managed using secret() expressions, preventing any hardcoding of AWS credentials within the workflow definition.
• The expiresIn parameter is explicitly set, ensuring time-limited access that automatically revokes permissions after one hour.
• The outputs of one task (the generated URLs) are directly accessible as variables in subsequent tasks, enabling seamless integration and testing within the same workflow.

Understanding the S3 Presigned URL Format

A presigned URL looks like a standard S3 object URL but with several additional query parameters appended. These parameters contain the authentication information S3 needs to verify the request.

Here is a breakdown of the key components: https://your-bucket.s3.your-region.amazonaws.com/object-key?query-parameters

The query parameters include:

• X-Amz-Algorithm: The signing algorithm used (e.g., AWS4-HMAC-SHA256).
• X-Amz-Credential: The access key ID of the creator, followed by the date, region, and service.
• X-Amz-Date: The timestamp when the URL was created.
• X-Amz-Expires: The lifetime of the URL in seconds.
• X-Amz-SignedHeaders: The HTTP headers that were included in the signature calculation (typically host).
• X-Amz-Signature: The final calculated signature, which validates the request.

This structure allows S3 to authenticate the request without the client needing to possess any secret keys.

Securing Your S3 Presigned URLs

While presigned URLs are a secure mechanism, their safety depends on proper implementation. Think of a presigned URL as a temporary bearer token: anyone who has it can use it.

Follow these best practices to maintain security:

• Apply the Principle of Least Privilege: The IAM role or user that generates the presigned URL should have the minimum permissions necessary. If a URL is only for downloading a specific object, the role should only have s3:GetObject permission on that specific object.
• Use the Shortest Possible Expiration Time: Set the Expires parameter to the shortest practical duration. If a user needs five minutes to upload a file, don’t generate a URL that lasts for an hour.
• Secure Distribution: Transmit presigned URLs over HTTPS to prevent them from being intercepted in transit. Be mindful of where they are logged or stored.
• Monitor and Log Access: Use S3 server access logging or AWS CloudTrail to monitor how presigned URLs are being used. This can help detect misuse or anomalous activity.
• Centralize Credential Management: Use a secrets management system, like the one built into Kestra, to handle the AWS credentials used for signing. This avoids hardcoding keys in your application code or workflow definitions.

Common Use Cases for S3 Presigned URLs

Presigned URLs are versatile and solve many common data access challenges in modern architectures.

• Securely Sharing Files: Provide temporary download links to external users, partners, or customers without adding them to your IAM policies or making objects public.
• Client-Side Uploads: Enable users to upload large files directly from a web or mobile application to your S3 bucket. This offloads traffic from your backend servers and simplifies your application architecture.
• Granting Temporary Access to Services: Allow a third-party service to temporarily read or write a specific object in your bucket as part of an integrated workflow.
• Multi-Tenant Data Exchange: In a multi-tenant application, you can generate presigned URLs to give tenants access only to their specific data objects, ensuring strict data isolation.

Ready to build robust, secure data workflows? Explore Kestra’s capabilities for declarative orchestration for modern data engineers.

Related concepts

• Kestra: Open Source Declarative Orchestration Platform
• Declarative Orchestration for Modern Data Engineers
• Data Engineering Resources: ETL, Orchestration & Pipeline Playbooks
• AWS Plugins: Generate a presigned EKS authentication token