Kafka Connectors Migration in 3 Hours

Michelin: Global, Distributed, Real-Time

Michelin is one of the world’s largest tire manufacturers, with operations spanning continents and decades of engineering leadership. Behind the scenes, their factories are increasingly software-defined—powered by event streaming and autonomous data infrastructure.

Each plant runs its own Kafka cluster to ensure operational independence. Across the globe, more than 60 Kafka clusters are live, with a central Confluent Cloud cluster serving cross-factory coordination. Kafka Connect acts as the data bridge—but operating it at scale proved difficult.

Originally, all connectors were deployed on a single Connect cluster, centrally managed via Kubernetes. As usage grew, that architecture began to crack. Rebalancing was painful. Deployments were global-impact. Offset syncing became fragile.

“We needed to move fast—but without asking application teams to rewrite anything. And most importantly, without re-replicating data.”

A Migration Workflow

The platform team made the call: the Connect infrastructure would be split by region—Europe, Americas, and Asia. This would improve operational boundaries, enhance offset syncing with their internal referential (ns4kafka), and set a solid foundation for scale.

But migrating connectors isn’t just “stop and recreate.” Kafka connectors store state in offset topics—not in consumer groups. The team needed a way to extract those offsets, recreate the connector in a stopped state, patch the offset data, and resume replication—seamlessly.

“People think connectors are just consumers—they’re not. Offsets live in topic messages. Replaying data was not an option.”

Enter Kestra: Supersonic Workflow Engineering

The team deployed Kestra via Helm using the PostgreSQL backend. In just three hours, they had built a testable, working workflow that:

• Stopped the connector
• Pulled its config and offsets via Kafka’s new KIP-875 API
• Recreated it on the target cluster using KIP-980 with initial_state: STOPPED
• Patched the offset topic
• Restarted the connector without reprocessing any data

All orchestrated in Kestra via declarative YAML, with full visibility across every task and API call.

“REST calls, offset transforms, parallel execution, retries—Kestra made all of it easy to string together, monitor, and evolve. And the built-in editor was a game-changer.”

Platform-Grade Observability, Developer-Grade Simplicity

Kestra gave Michelin exactly what they needed: a control plane for migration, not a tangle of fragile scripts. Engineers could trigger workflows with input parameters, migrate dozens of connectors in parallel, and inspect output with real-time Gantt charts and task-by-task logs.

Application teams? Barely involved. They only reviewed results—no rewrites, no downtime.

“We didn’t have to involve app teams. No connector rewrites. No duplication. No surprises. Just a clean migration with full traceability.”

Laying the Groundwork for Resilient Data Infrastructure

For a global manufacturer like Michelin, orchestrating critical infrastructure isn’t just about uptime—it’s about confidence. With Kestra, the team now has a reusable, observable, and fast framework to manage change at scale—without trade-offs. A big thank you to Alexis Souquiere for sharing his story with us. If you want to learn more, you can read his full blogpost of the migration on Michelin's blog.

“Everything we needed was already in Kestra—and in Kafka, since 3.7.0. We just had to connect the dots. Now we spend less time on orchestration and more time building the platform our teams rely on.”