Data · AI/ML
Pipeline orchestration
Schedule EoH ETL · Trino queries · scripts as one DAG. Even on failure, idempotent re-run keeps results stable.
PADION
flowkeeper · Workflow Orchestration
DAG · Schedule · Idempotent · Audit
What runs, when, in what order —
deterministically
Define work as a DAG and run it on schedule. Partial failures come back through idempotent re-run, host failures fail over, and every execution is recorded tamper-proof.
Workflow DAG (example)
Parallel run → conditional branch → deterministic re-run
What is
Why a workflow orchestrator
As batch jobs grow, what depends on what, when it must run, and where to resume after a failure quickly becomes operational risk. Schedulers and scripts alone can't consistently guarantee ordering, state, and re-run.
PADION flowkeeper defines work as a DAG in a Project · Workflow · Task hierarchy and executes it by resolving dependencies, parallelism and conditional branches. The essence is three things — dependency resolution · state tracking · idempotent re-run.
If EoH is the ETL engine that moves the data, flowkeeper is the orchestrator that controls when and in what order any job (ETL included) runs. SQL · HTTP · Trino · scripts all live in one DAG.
Key Features
Six essentials
DAG engine — order · parallel · condition
Dependencies run sequentially, independent tasks in parallel — with on-success / on-failure / always / expression-based branching, all defined as a graph.
Crash-safe · deterministic re-run
Stuck tasks auto-recover, and partial failures come back through idempotent re-run. Re-run as many times as you like — the result is the same.
Host affinity · failover
Pin workflows to a host/pool; on host failure choose HALT or FAILOVER. Concurrency caps prevent load spikes.
Auth · tamper-evident audit
Authenticate via gatekeeper OIDC and govern with role-based access (RBAC). Every action is sealed in a monthly HMAC hash chain to detect later deletion or tampering.
Embedded web UI
Execution history · per-run Gantt timeline · drag-and-drop DAG editor · schedules · audit-chain verify — all in one console. Azkaban workflow import supported.
Multiple task types
Compose SQL · HTTP · Trino queries · scripts in one DAG. EoH ETL is just one kind of job flowkeeper drives.
How it Works
Define → Schedule → Recover
1
Define workflow
Compose work as a Project · Workflow · Task hierarchy. Express dependencies · parallelism · conditional branches as a DAG.
2
Schedule · execute
Run automatically on cron schedules. Track execution state with pause · resume · cancel.
3
Recover · audit
Partial failures re-run idempotently from where they stopped. Every history is kept as tamper-evident audit.
Use Cases
Real-world scenarios
Finance · Public
Batch ops in isolated networks
A coordinator/worker split minimizes credential exposure on workers, keeping large batch operations governable and auditable.
Tech Spec
Technical specs
| Execution model | DAG (order · parallel · conditional branch) |
|---|---|
| Scheduling | cron expressions · dependency triggers |
| Re-run · recovery | Idempotent re-run · snapshot compensation · auto-recovery of stuck tasks |
| Availability | Host affinity · HALT / FAILOVER · concurrency caps |
| Task types | SQL · HTTP · Trino · script |
| Auth · audit | gatekeeper OIDC · RBAC · monthly HMAC hash-chain audit |
| Migration | Azkaban (Flow 2.0 · 1.0) import |
| PADION integration | Drives EoH · Trino jobs (auth delegated to gatekeeper) |
In the PADION Flow
Workflow step — 05
After permissions are defined, flowkeeper controls the order and re-run of pipeline jobs, and EoH performs the ingest/transform as directed.
Not scheduling,
deterministic orchestration.
PoC · Azkaban migration · pipeline integration.