Mecku: Visual Data Pipeline Orchestration Platform
A drag-and-drop pipeline builder that lets analysts compose distributed ETL jobs across ten-plus data sources without writing Spark code.
The Problem
Building production-grade data pipelines required deep Spark and infrastructure expertise, which limited who could ship pipelines. Delivery cycles were long and brittle because each new pipeline repeated the same engineering work. Business teams could not move quickly from request to execution.
Data fragmentation compounded the skill barrier. Sources spread across cloud storage, operational databases, and data warehouses each carry their own integration API, so even a straightforward join across two systems became an engineering project measured in weeks.
What We Built
We built a visual pipeline orchestration platform with a node-based interface and execution back-end for scalable runs. Users can define, connect, and manage data flows through a guided UI while engineering retains control of runtime behavior. This reduced custom pipeline implementation work and improved repeatability.
Pipelines are composed as a graph of source, transformation, and sink nodes on a canvas. That graph compiles into a distributed execution plan that respects stage dependencies, so what an analyst designs visually runs as a real, fault-tolerant distributed job rather than a toy workflow.
Visual node-based designer
Analysts compose pipelines by dragging source, transformation, and sink nodes onto a canvas and configuring each through property panels, with the layout persisted for later editing.
Broad multi-source connectivity
Built-in connectors span cloud object storage, document and relational databases, and data warehouses, so cross-system pipelines do not require learning each integration API.
Rich transformation library
Filter, merge, union, pivot, SQL query, formula, find-and-replace, type conversion, and date operations cover the majority of pipeline logic without dropping into code.
Distributed DAG execution
The visual graph compiles to a dependency-respecting execution plan running on a distributed cluster, with parallel stage execution and fault tolerance handled by the platform.
How It Works in Practice
A marketing team that needs to combine customer records from one database with transaction data from another, transform the result, and load it into a warehouse for reporting.
- 1
Add the sources
Two source nodes go on the canvas—one per system—each configured with its connection details and target collection or table through a property panel.
- 2
Join and transform
A merge node connects both sources on a shared key, then transformation nodes rank the top customers by transaction value and compute aggregate spend per customer.
- 3
Define the destination
A sink node targets the reporting warehouse with the destination database, schema, and table specified—no output-format code required.
- 4
Run and observe
Execution runs the stages in dependency order across the cluster, reporting real-time progress with sample data visible at each stage so problems surface during the run.
Engineering Approach
The canvas compiles to a real execution plan
The visual graph is not a wrapper around a script—it translates into a dependency-ordered distributed job, so what an analyst designs runs with the same fault tolerance as a hand-written pipeline.
Separation of design and runtime
A configuration service owns pipeline definitions and canvas state while a separate execution manager coordinates distributed runs, letting engineering tune runtime behavior without touching the authoring experience.
Connectors as the extension point
Sources, transformations, and sinks are pluggable node types, so supporting a new system or operation extends the existing library rather than requiring changes to the execution engine.
Have a similar problem?
Tell us what you're working on and we'll tell you honestly whether we can help.
