Pipelines
A pipeline is the core unit of work in KMP: a named, versioned data-processing workflow that moves data from a source connector, through zero or more transform steps, to a destination connector. Every pipeline runs as a sequence of events flowing through that path.
Anatomy of a pipeline
Every pipeline has three parts:
- Source — the connector that emits events into the pipeline (a database, a queue, an upload, a webhook).
- Transforms — an ordered list of steps that reshape, filter, or enrich events as they pass through. Transforms are optional; a pipeline with none is a pass-through copy.
- Destination — the connector that events are written to once transforms complete.
You describe the desired source, transforms, and destination; KMP schedules and retries the runs. You never manage worker processes directly.
Pipeline state
A pipeline is always in one of three states:
draft— configured but not yet processing events.active— running on its configured schedule or trigger.paused— temporarily stopped; resumes from the last checkpoint.
Versioning
Pipeline configuration is versioned. Publishing a new version does not affect events already in flight on the previous version — in-flight runs finish on the version they started on. This matters most when a pipeline definition is part of a larger migration; see the Migration guide for an example of retiring an old pipeline version safely.
Next steps
Ready to build one? Start with the Building your first pipeline guide, or jump straight to the Pipelines endpoint in the API reference.