Cross-cloud incremental backup of Dropbox to Amazon S3

Keep a durable, off-platform copy of a Dropbox folder in an Amazon S3 bucket with an automated nightly backup. This flow is incremental: each run lists the Dropbox folder, then downloads and uploads only the files that changed since the last successful run, tracked through a KV store watermark. That keeps Dropbox egress and S3 PUT costs proportional to what actually changed and stops the job slowing down as the folder grows, solving the problem of brittle, full-copy backup scripts that re-transfer everything every night and silently skip files when they fail. It pairs the Dropbox plugin with the AWS S3 plugin for a cross-cloud, schedule-driven disaster-recovery copy.

How it works

1. list_dropbox_files (io.kestra.plugin.dropbox.files.List) recursively lists entries under dropbox_folder, capped at the max_files input (default 1000) with fetchType: FETCH so the rows are available to downstream tasks.
2. backup_each_file (io.kestra.plugin.core.flow.ForEach) iterates over outputs.list_dropbox_files.rows with a concurrencyLimit of 4.
3. backup_if_changed (io.kestra.plugin.core.flow.If) keeps only entries whose type is file (a recursive listing also returns folders, which Download cannot fetch) and whose clientModified timestamp is newer than vars.watermark.
4. For each matching file, download_from_dropbox (io.kestra.plugin.dropbox.files.Download) stages it into Kestra internal storage, then upload_to_s3 (io.kestra.plugin.aws.s3.Upload) writes it to s3_bucket under a key that mirrors the Dropbox path beneath s3_prefix using STANDARD storage class.
5. update_watermark (io.kestra.plugin.core.kv.Set) records the run's start time to the DROPBOX_BACKUP_WATERMARK KV key, but only after every changed file uploaded, so a mid-run failure re-tries the same window next time.
6. A Schedule trigger (nightly_backup) fires the backup at 02:00 UTC daily, and an alert_failure error handler posts a Slack message if the run does not complete.

What you get

• An incremental, cost-aware backup that transfers only new and modified files.
• At-least-once delivery: the watermark advances only on full success, so failures never silently skip files.
• Per-task retries (5 attempts, constant 10s) on listing, download, and upload.
• Proactive Slack alerting when a nightly backup fails.

Who it's for

• Platform and infrastructure engineers who need an off-platform disaster-recovery copy of shared Dropbox content.
• Operations and IT teams consolidating SaaS file storage into S3 for retention or compliance.
• Data engineers who want Dropbox documents landed in S3 for downstream processing.

Why orchestrate this with Kestra

Neither Dropbox nor S3 ships a scheduler that can list one cloud, conditionally copy to another, and track incremental state across runs. Kestra fills that gap: a declarative YAML flow with an event or Schedule trigger, built-in per-task retries, a ForEach for bounded parallelism, an If guard for change detection, a persistent KV watermark for incremental state, and full execution lineage and logs over every transferred file. The result is observable, restartable, and version-controlled, instead of a cron-driven shell script no one can debug.

Prerequisites

• A Kestra instance with the Dropbox, AWS, and Slack plugins installed.
• A Dropbox app access token with read access to the source folder.
• AWS credentials (or an attached IAM role) with s3:PutObject on the target prefix.
• An existing S3 bucket to receive the backup copies.

Secrets

• DROPBOX_ACCESS_TOKEN: Dropbox API access token used to list and download files.
• AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID: AWS credentials for the S3 upload. Prefer an IAM role or instance profile and omit static keys; if you must use keys, scope the policy to arn:aws:s3:::<bucket>/<prefix>/*.
• AWS_DEFAULT_REGION: AWS region of the destination S3 bucket, for example eu-west-1.
• SLACK_WEBHOOK: Slack incoming webhook URL used by the failure alert.

Never hardcode any of these values.

Quick start

1. Add the DROPBOX_ACCESS_TOKEN, AWS_ACCESS_KEY_ID, AWS_SECRET_KEY_ID, AWS_DEFAULT_REGION, and SLACK_WEBHOOK secrets to your namespace.
2. Set the dropbox_folder, s3_bucket, s3_prefix, and max_files inputs.
3. Enable S3 versioning on the bucket for delete and overwrite protection, and add a lifecycle rule to transition older versions to STANDARD_IA or GLACIER after N days.
4. Enable the flow and confirm the first nightly run lands objects in your bucket. The first run copies everything (the watermark starts at the Unix epoch); later runs copy only changes.

How to extend

• Swap the Schedule trigger for a higher frequency, or trigger the flow from an upstream event.
• Partition very large folders into sub-folders and run one flow per sub-folder to stay under max_files.
• Add a post-upload notification or manifest task to log the set of objects written each night.
• Replace S3 with another object store, or fan out to a second region for cross-region redundancy.

Links

• Dropbox plugin: https://kestra.io/plugins/plugin-dropbox
• AWS plugin: https://kestra.io/plugins/plugin-aws
• KV Store: https://kestra.io/docs/concepts/kv-store