Apache Wayang, the Wunderkind of data processing?

Apache Wayang is an open source cross-platform data processing framework and an Apache top-level project since November 2025. You write a pipeline once, and its cost-based optimizer decides which engine runs each step, for example Postgres, Spark, Flink or a plain Java process, and moves only the data the next step needs. It started as the Rheem research project at the Qatar Computing Research Institute, with papers at VLDB, ICDE and SIGMOD. This post answers whether Wayang is really a wunderkind, explains how it differs from Spark and Trino, shows what the official benchmarks say, and lists where it does not help.

Wunderkind is a big word. It means a child who is brilliant early. So is Apache Wayang the wunderkind of data processing? Honestly, no. Wayang is not young and it did not get famous overnight. It took about ten years of research, papers, an Apache incubation and a lot of quiet work on the mailing list. What it has is a good idea that most data teams need and almost nobody else builds: one plan for a pipeline, and the right engine for each step.

I was on the first podling team when Wayang entered the Apache Incubator in December 2020, and I sit on the Wayang PMC today. So read this with that in mind. I will stick to what the project, the papers and the Apache records say.

Project: wayang.apache.org. Code at github.com/apache/wayang, Apache License 2.0, Java and Scala APIs plus SQL, with a Python API that covers fewer operators for now.

The problem Wayang solves

Look at a normal data pipeline in a company today. Customer and order rows sit in Postgres. Events and history sit in a data lake on S3 or HDFS. The heavy joins run in Spark or Flink. The model trains in TensorFlow or PyTorch. Every arrow between those boxes is glue code, an export job, a copy of the data and one more thing that breaks at night.

The usual fix is to pick one engine and copy everything into it. That works, but you pay for it. You pay egress to move the data, storage for every copy, and engineers to keep each export alive. And some data is not allowed to move at all, because it has to stay in one country, one plant or one closed network.

Wayang takes another route. You describe the pipeline once against Wayang's API. Wayang turns it into a plan that is not tied to any engine. Then its optimizer places every step on one of the engines you registered, close to where the data already is, and only ships what the next step needs.

I wrote before that data processing does not belong in the message broker. Kafka moves data, engines process it. Wayang is the next layer up from that idea. It does not process data itself. It decides which engine should.

Where Wayang comes from

Wayang began as Rheem, a research project at the Qatar Computing Research Institute (QCRI). The first Rheem papers came out in 2016, a vision paper at EDBT and a demo at SIGMOD. The main paper, RHEEM, Enabling Cross-Platform Data Processing, appeared in the Proceedings of the VLDB Endowment in 2018. It describes the cost-based optimizer that also prices data movement between engines, and that idea is still the core of Wayang.

More research followed. The Wayang site lists these as the main publications:

RHEEM, Enabling Cross-Platform Data Processing. Proc. VLDB Endow. 11(11), 2018
RHEEMix in the data jungle, a cost-based optimizer for cross-platform systems. VLDB Journal 29(6), 2020
ML-based Cross-Platform Query Optimization. ICDE 2020
Apache Wayang, A Unified Data Analytics Framework. SIGMOD Record 52(3), 2023
Apache Wayang in Action. SIGMOD Conference Companion 2025

Then came the Apache part. Wayang entered the Apache Incubator on December 16, 2020. It shipped 1.0.0 in early 2025 and 1.1.0 in mid 2025. The Incubator voted to graduate it on November 4, 2025, and the ASF board established Apache Wayang as a top-level project on November 19, 2025. The ASF announced it on December 3, 2025, together with Apache Artemis, in its new top-level projects post. Today the project has 18 committers and 9 PMC members, and Zoi Kaoudi, one of the original Rheem authors, chairs the PMC.

So it is not a wunderkind. It is a research idea that grew up slowly and in public, which in my experience is the better way for infrastructure to grow up.

From my side on the podling and later the PMC, the hard part was never the code. It was building a community around what started as a research project, and getting everyone used to the Apache way of working: decisions on the mailing list, releases voted in the open, and committers who earn their seat through contributions. Once that was in place, everything else went smoothly.

How Wayang decides which engine runs each step

The Wayang site describes three layers. Your application logic goes in at the top. The core turns it into a Wayang plan. The platform layer turns that plan into an execution plan where each operator runs on a specific engine.

The optimizer does the interesting work. It first "inflates" the plan by adding every possible engine-specific version of each operator. Then it attaches costs, both for running each operator and for moving intermediate data from one engine to another. By default those are linear cost formulas, and you can plug in your own model, for example an ML-based one. Finally it picks the cheapest plan. The cost can be runtime, money or even energy.

Engines supported today, according to the README, are Java Streams, Apache Spark, Apache Flink, Apache Giraph, PostgreSQL, SQLite, Apache Kafka and TensorFlow. More can plug in.

In code it looks like this. This is the pattern from the Wayang README. You register the engines you have, and the optimizer chooses per step:

// Register both platforms. Wayang's optimizer decides which to use per step.
WayangContext wayang = new WayangContext(new Configuration())
        .withPlugin(Java.basicPlugin())
        .withPlugin(Spark.basicPlugin());

If a step must stay on one engine, for example because the data cannot leave a database, you pin it with withTargetPlatform(...) on that operator. The optimizer then plans the rest around your decision.

One practical lesson. On tiny test data the optimizer will keep everything local, because Spark's startup cost is not worth it for a small file. People think Wayang ignores Spark. It does not. That is the optimizer doing its job. Cross-engine splits show up once the data is big enough to justify them.

Wayang vs Spark vs Trino

The question I hear most is "why not just use Spark?" or "we already have Trino". Fair question. They solve different problems.

Spark is a compute engine. It runs SQL, streaming, ML and graph jobs on a cluster. If your rows live in Postgres and your history lives in a lake, Spark reads the Postgres data into its executors and does the work there. The data is copied into the cluster first.

Trino is a distributed SQL query engine. It runs one SQL query across many sources through connectors. Its own docs say Trino workers fetch data from connectors and exchange intermediate data with each other. So the data is pulled to Trino's workers at query time, and the scope is SQL analytics. Trino's docs also say plainly that it is not a general-purpose database and not built for OLTP.

Wayang is neither. It is an optimizer above your engines. It plans the whole pipeline, SQL or not, and lets Spark, Flink, Postgres or a Java process do the actual work.

	Apache Spark	Trino	Apache Wayang
What it is	Compute engine on a cluster	Distributed SQL query engine	Optimizer above your engines
Workloads	SQL, streaming, ML, graph	SQL analytics and reports	Pipelines that mix SQL, ML and more
Where data goes	Copied into the Spark cluster	Pulled to Trino workers at query time	Stays where it is, only what the next step needs moves
What the optimizer decides	How a job runs inside Spark	How one SQL query runs	Which engine runs each step

So my advice is simple. Keep Spark and keep Trino. Your analysts like Trino and your data engineers know Spark. Wayang does not replace them. It decides which engine runs each step, and Spark is one of the engines it can use.

The same thinking is behind my post on SQL on streaming data without a streaming engine. Use the tool that fits the question, not the one that happens to be running.

What the official benchmarks say

I will only quote the project's own benchmark page here. Read it yourself for the setup and the charts.

Federated SQL with TPC-H query 5. The lineitem and orders tables sat in HDFS, customer, supplier and region in Postgres, and nation in S3 or the local file system. The baselines were Spark and Postgres, with all data moved into them first. The page says "Wayang significantly outperformed Postgres while retaining a runtime very close to Spark." The detail that matters: "the time required to transfer data to Spark or Postgres was excluded from the results", and "for Spark to function, we needed twice the time to extract datasets from Postgres and transfer them to Spark." Wayang ran the selections and projections in Postgres and the big join in Spark, without anyone telling it where to run what.

Machine learning with stochastic gradient descent. Compared with Spark MLlib and IBM SystemML, the page says "Wayang outperforms both baselines by more than an order of magnitude in runtime performance for large datasets." The trick is that Spark does the preprocessing, and once the data has shrunk, a single Java process takes over the gradient steps.

WordCount from 1 GB to 800 GB. With Spark, Flink and a single-node Java program available, the page reports Wayang "consistently selecting the fastest available platform for various dataset sizes".

None of these results says Wayang is faster than every engine at everything. They say the optimizer picks good placements, and that skipping the copy step is where much of the time goes.

Where Wayang does not help

Wayang removes glue code. It does not remove physics. Every switch between engines costs serialization, data movement and a new scheduling cycle. I wrote a full post on the performance cost of cross-platform execution in Wayang, so here is the short version:

One engine, one data store. If all your data is already in one lake and Spark does everything, Wayang adds a layer and little else.
Bad cost estimates. The optimizer is only as good as its cost model. If the model does not fit your cluster, the plan will not either. Tune it with real runs.
Too many engine switches. A plan that jumps between engines at every step will be slow. Group steps on the same engine where you can.
Setup friction. The README asks for Java 17 with extra JVM flags, Spark 3.4.4 with Scala 2.12 and Hadoop 3. Plan time for the first clean install.
Python first teams. The Python API covers fewer operators than Java and Scala for now.

The case I see most in production is model training on old data silos. The training data sits partly in SQL databases that have been running for years and partly in Databricks. The usual answer is to copy everything into one place first and then train. With Wayang you write the pipeline once, the filters and joins run where the data already lives, and only the reduced training set moves to the engine that trains the model.

So is Wayang the wunderkind of data processing

No, and that is the good news. A wunderkind is surprising and fragile. Wayang is ten years of research that went through peer review at VLDB, ICDE and SIGMOD and then through the Apache Incubator, and it is now a top-level project with its own PMC. What it offers is not a faster engine. It is a planner that lets the engines you already pay for each do the part they are good at, and stops you from copying everything into one of them first.

If your pipelines cross Postgres, a lake, Spark or Flink and an ML stack, start small. Take one pipeline that has the most glue code, register two engines, and look at the plan Wayang picks. My older guide to multi-engine data processing with Apache Wayang walks through the bigger picture.

If you want to run Wayang in production with tested releases, hardening, support and help moving existing Spark and Flink jobs, Scalytics offers a paid Wayang add-on for Lascaris. The details are on the Apache Wayang in production page. Wayang itself stays free under the Apache License 2.0 and is run by the Apache community.

Frequently asked questions

What is Apache Wayang?

An open source cross-platform data processing framework and Apache top-level project. You write a pipeline once, and its optimizer runs each step on the engine that fits, such as Postgres, Spark, Flink or a Java process.

Does Wayang replace Spark or Flink?

No. Spark and Flink are engines. Wayang sits above them and decides which one runs each step.

How is Wayang different from Trino?

Trino runs SQL queries across sources and processes the data on its own workers. Wayang plans whole pipelines, SQL or not, and lets the underlying engines do the processing.

When did Apache Wayang become a top-level project?

The ASF board established it on November 19, 2025, after it entered the Apache Incubator in December 2020.

Where did Wayang come from?

It started as Rheem, a research project at the Qatar Computing Research Institute, published at VLDB in 2018 with later papers at ICDE and SIGMOD.

Is Apache Wayang free?

Yes. It is licensed under the Apache License 2.0. Commercial support exists separately, for example the Scalytics Wayang add-on for Lascaris.

Sources

Need help with distributed systems, backend engineering or data platforms? See my consulting services.