0.1.0-ALPHA IS HERE

Your data.
In motion.
In your hands.

Copy tables, changes, and events on a schedule or continuously. Transform with Spark, Flink, and dbt, and give exceptions a path to recovery. Open source. Self-hosted. In your hands.

Apache 2.0/Built in the open

THE OPENETL DATA WORKSHOP
a little mixed up?TRANSFORMLet’s make something of it.YOUR SOURCESYOUR ENGINES. YOUR INFRASTRUCTURE.++
Extract. Transform. Load. And transform again.Meet your guide
THE OPEN WAY TO MOVE DATA.Follow the flow EXTRACT → TRANSFORM → LOAD → TRANSFORM
YOUR INFRASTRUCTUREYOUR PIPELINESYOUR POSSIBILITIESNO BLACK BOX REQUIRED.
+ INSTALLATION & QUICK START

Your first pipeline.
Starts here.

Install the CLI. Start the released stack. Then follow the quick start to move your first rows and see them arrive. Allow about 15 minutes for the full walkthrough.

macOS or Linux · amd64 / arm64Docker Compose 2.24+curl + openssl · 2 GB disk
Deploying for your team? Self-hosting guide
01

Install the CLI

v0.1.0-alpha

One command downloads the release and checks its SHA-256 checksum.

MACOS / LINUX
curl -fsSL https://raw.githubusercontent.com/openetl-io/openetl/main/install.sh | OPENETL_VERSION=0.1.0-alpha sh
Verify with openetl version

Installs to ~/.local/bin. Add it to your PATH if prompted.

Manual install & verification
02

Start OpenETL locally

In a new directory, run the same release with Docker Compose.

DOCKER COMPOSE
mkdir openetl && cd openetl
curl -fsSLO https://raw.githubusercontent.com/openetl-io/openetl/v0.1.0-alpha/deploy/compose/compose.yml
cat > .env <<EOF
OPENETL_IMAGE=ghcr.io/openetl-io/openetl:0.1.0-alpha
POSTGRES_PASSWORD=$(openssl rand -hex 16)
OPENETL_SECRETS_KEY=base64key://$(openssl rand -base64 32 | tr '+/' '-_')
EOF
chmod 600 .env
docker compose up -d
Console at localhost:8080 · sign-in setup below

First start downloads the images. The quick start covers sign-in and connecting the CLI.

03

Make the first connection.

Sign in, install a connector, sync generated rows, and verify the result.

Follow the quick start

Alpha release. APIs, pipeline format, and connector protocol can change between versions.

+ A SMALL GUIDE TO A BIGGER PICTURE

Every bit has a story.
Let’s follow one.

Meet your guide: the OpenETL mark, with a little life of its own. Follow a batch from mixed inputs to its next big thing.

01 / COLLECT

It starts a little mixed up.

Tables here. Events there. A stream of changes that never stops. Bring them onto the same production line, on a schedule or continuously.

TablesCDCEvent streams

Apples and oranges are our stand-ins for your data.

02 / TRANSFORM

Give the heavy lifting to your engines.

Send scheduled workloads through Spark or continuous streams through Flink. Your transformation code runs on your own clusters, coordinated by OpenETL.

Apache SparkApache FlinkYour clusters

Distributed processing. Familiar tools. One pipeline.

03 / PROTECT

A bad value shouldn’t spoil the whole batch.

Hold a column, a row, or a single cell when values stop fitting. Keep compatible data moving where your destination supports it, and give the exception a safe place to wait.

Column holdsRow quarantineCell quarantine

Repair the cause. Approve a release. Apply it on the next run.

04 / DELIVER

Land it. Shape it. Keep going.

Deliver into your database, lake, or stream. Run dbt after a compatible scheduled sync. Recover from the last checkpoint when a run fails, and release repaired values back into the flow.

Post-sync dbtCheckpointsRepair & release

Delivery guarantees are checked for each source and destination.

See how it fits together
THE DATA WORKSHOP
THE TRANSFORMATION FLOORSparkFlinkQUARANTINEheld safely for repairRELEASEDESTINATIONyour destinationAFTER A SCHEDULED SYNCdbtbuild / run / testMIXED INPUTSApples. Oranges. Let’s sort this out.Tables, changes & events → your pipeline
Scroll to follow the journeyIllustrated workflow
+ YOUR WHOLE ETL(T) STACK

Extract. Transform. Load.
And transform again.

Your engines, coordinated. Transform on the way with Spark or Flink, then build models with dbt after a compatible scheduled sync.

TRANSFORM ON THE WAY

Apache Spark

Batch, with all the muscle.

Run your transformation code between source and destination on your own Spark cluster. DataFrames in. Your outputs out.

Scheduled pipelines · In-flight + post-syncExplore Spark integration
TRANSFORM AS IT HAPPENS

Apache Flink

Streaming that keeps its state.

Bring your Flink jobs into continuous pipelines. Transform DataStreams on your own cluster, with checkpoints coordinated by OpenETL.

Continuous pipelines · In-flightExplore Flink integration
TRANSFORM AFTER IT LANDS

dbt

From loaded to ready to use.

Run your dbt project after a scheduled sync commits. Build, run, or test models against a compatible destination.

Scheduled pipelines · Compatible destinationsExplore dbt integration

Easy to start. Room to grow.

Start with a simple pipeline. Bring in the rest of your stack when you need it.

PIPELINE EXPLORERInteractive illustration
SOURCESource systemTables & snapshots
sync
DESTINATIONDestination systemScheduled delivery
Every hourYour infrastructure
01

A repeatable sync, on your clock.
Define the pipeline. Review the plan. Apply it.

pipeline.yaml
apiVersion: openetl.io/v1alpha1
kind: Pipeline
metadata:
  name: my-pipeline
spec:
  schedule: { every: 1h }
  source: { ref: my-source }
  destination:
    ref: my-destination
    namespace: analytics

Illustrative YAML. Configure the source and destination references before applying.See the complete example

+ EXCEPTIONS GET A WAY FORWARD

One bad cell.
A whole lot worth keeping.

Column, row, or cell-level quarantine. Isolate what needs attention, repair the cause, and release it back into the flow.

Hold it. Fix it.
Pick up from there.

A changed schema doesn’t have to mean starting over. See what’s held and why, choose the repair, and approve what happens next.

01Inspect the hold02Repair & approve release03Apply on the next run

A release takes effect on the next pipeline run. Resume a paused pipeline or stream so it can run. Cell holds require compatible destination support.

Read the recovery walkthrough
THE EXCEPTION DESKInteractive illustration
Just the cell.

One value waits. The other fields in its row can land. Requires a primary key and a destination that supports patching held values.

Illustrative destination: orders amount: int32
id PKfruitamount
101Apples100
102Orangesheld — quarantined
103Apples240
Value safely held2147483648 exceeds int32
Repair the destination’s type first.
+ LESS BLACK BOX. MORE BUILDING BLOCK.

Built for the people
behind the pipelines.

Your data infrastructure should work the way you do. Inspectable, adaptable, and under your control.

01

Your stack. Your control.

Run OpenETL on your own infrastructure. Keep your data, credentials, and operations in your hands.

SELF-HOSTED BY DESIGN
02

Pipelines belong in Git.

Define pipelines in YAML. Review a plan before applying changes. Bring the way you ship software to the way you move data.

DECLARE. PLAN. APPLY.
03

Know what actually landed.

Inspect freshness and failures in the console. Send metrics and traces to tools that speak OpenTelemetry.

VISIBILITY BUILT IN
04

Put the work where it belongs.

Keep orchestration in the control plane. Run your data plane where it can reach your systems, and bring your own transformation clusters.

CONTROL PLANE + DATA PLANE
↳

Promises, with the details attached. Exactly-once delivery depends on the source and destination. OpenETL checks each stream’s guarantees before running the pipeline.

Understand delivery
+ MEET YOUR CONNECTIONS

Different systems.
Common ground.

Connect databases, streams, open table formats, and object storage. Install official packages from the registry, bring a community connector, or build one for your systems.

Explore the connector repo
openetl connector catalog

In the console: Connections → Connectors → Catalog. Use the install command shown for your package. Installation & versions

Independently versioned packages. Official alpha packages currently match 0.1.0-alpha.
+ A FEW GOOD QUESTIONS

Before you
dive in.

Where is OpenETL in its release journey?

OpenETL 0.1.0-alpha is available now. Install the CLI and run the published release with Docker Compose, or deploy it on Kubernetes. The pipeline format, APIs, and connector protocol can change between alpha releases. Follow GitHub releases and use the docs for your version.

Is it really open source?

Yes. OpenETL and its official connectors are licensed under Apache 2.0. You can inspect, modify, and self-host the code. Start with the core repository.

Can I run batch and streaming pipelines?

Yes. The same platform runs pipelines on a schedule or continuously. The connector and source configuration determine how data is read. Connect databases, event streams, open table formats, and object storage using the read and write modes supported by each connector.

Where do Spark, Flink, and dbt fit?

Spark transforms data in scheduled pipelines on your own cluster, and can also run after a sync. Flink transforms continuous streams, with its checkpoints coordinated by OpenETL. dbt runs after a scheduled sync commits to a compatible destination. See the integration guide for supported targets and setup.

What happens when only part of the data stops fitting?

Schema policies and connector capabilities determine whether OpenETL can hold a column, quarantine a row, or hold a single cell while compatible data keeps moving. Cell holds require a primary key and a destination that can patch values. Repair the cause and approve the release; the next run applies it. Resume any paused pipeline or stream so it can run. The recovery walkthrough shows a complete example.

Do connectors release with OpenETL?

Connectors are installed as versioned packages, separately from OpenETL. In this alpha, the official connectors share its 0.1.0-alpha version. List them with openetl connector catalog, then use the install command it shows. Pin versions for reviewed deployments and check compatibility before upgrading. See the connector installation guide.

Does every pipeline deliver exactly once?

Delivery guarantees depend on both ends of the pipeline. OpenETL checks what each stream can provide and refuses a pipeline that cannot meet its requested guarantee. Read the connector documentation for the conditions, supported modes, and limitations.

Where can I ask questions or share what I’m building?

Join the OpenETL Slack community to ask questions, discuss connectors, and share your pipelines. For bugs and feature requests, open a GitHub issue so the discussion is easy to follow.

How can I help shape the alpha?

Try a pipeline, report what you find, or contribute a fix or connector. The contribution guide covers the development setup and how to send a pull request.

OPEN SOURCE. OPEN POSSIBILITIES.

Good data moves.
Let’s get it going.