Office of Kaiju Affairs

Building your first pipeline

This guide walks through creating a pipeline end to end: a source connector, a destination connector, and the pipeline that ties them together.

Before you start

You'll need an API key. See Authentication in Getting Started if you don't have one yet.

1. Create a source connector

We'll pull orders from Postgres. See Connectors for the full catalog of types.

create-source.sh bash 
curl -X POST https://api.kmp.kaiju.go.example/v2/connectors \
  -H "Authorization: Bearer $KMP_API_KEY" \
  -d '{"type":"postgres","direction":"source","config":{"connection_string":"postgres://..."}}'

2. Create a destination connector

We'll write to Snowflake.

create-destination.sh bash 
curl -X POST https://api.kmp.kaiju.go.example/v2/connectors \
  -H "Authorization: Bearer $KMP_API_KEY" \
  -d '{"type":"snowflake","direction":"destination","config":{"account":"..."}}'

3. Create the pipeline

Full request/response details are in the Pipelines endpoint reference.

create-pipeline.sh bash 
curl -X POST https://api.kmp.kaiju.go.example/v2/pipelines \
  -H "Authorization: Bearer $KMP_API_KEY" \
  -d '{"name":"ingest-orders","source_connector_id":"conn_pg_orders","destination_connector_id":"conn_snowflake_dw"}'

The pipeline starts in draft state. Activate it once you're ready to start processing events.

Next steps

Now that events are flowing, read Handling errors & retries to see what happens when a write fails, or Monitoring & alerts to watch it run.