Software:Apache DataFusion

From HandWiki
Apache DataFusion
Developer(s)Apache Software Foundation
Written inRust
TypeQuery engine
LicenseApache License
Websitedatafusion.apache.org

Apache DataFusion is an open-source, extensible analytical query engine written in Rust, built on Apache Arrow's columnar memory format.[1][2] It provides SQL and DataFrame interfaces for analytical query execution and is designed to be used as a library by developers building databases, query engines, and analytical tools, rather than as a standalone database server.[1][2] The project originated in 2017, was donated to the Apache Arrow project in 2019, and became a top-level project of the Apache Software Foundation in 2024.[3][4] As of March 2026, DataFusion exceeded one million monthly downloads on crates.io.[5]

History

DataFusion originally authored by Andy Grove starting in 2017. It was donated to the Apache Arrow Project in February 2019.[3] In 2024, a paper describing DataFusion was accepted to the industry track of the ACM SIGMOD conference.[6][1] In April 2024, the project graduated from Apache Arrow and became a top-level Apache project.[4]

Features

DataFusion is a fast, extensible query engine for building data systems. It provides a SQL interface and a DataFrame API for constructing queries programmatically, a query planner and rule-based optimizer, and a multithreaded vectorized execution engine that processes data in columnar batches rather than row by row.[1][2]

The engine reads common analytical file formats natively, including Apache Parquet, CSV, JSON, Avro, and Arrow IPC, and uses Apache Arrow's columnar memory format throughout execution, avoiding serialization overhead between stages.[1]

DataFusion is designed for in-process embedding: it runs within the host application's process rather than as a separate server, using threads for parallel query execution. Its extension points allow downstream systems to add user-defined functions, custom data sources, custom query languages, and new optimizer rules, enabling developers to build specialized database systems on top of DataFusion's planning and execution components without reimplementing them.[1][2]

DataFusion is frequently compared with other columnar analytical systems including DuckDB, Polars, and Velox, but these systems differ significantly in scope and intended use.[7]

Adoption and reception

DataFusion has been adopted across a range of analytics and database products. Cloudflare used DataFusion in its Log Explorer product to execute SQL queries over log data stored in Cloudflare R2.[8] Palantir Lightweight Pipelines are powered by DataFusion.[9][10] InfluxDB 3.0 uses DataFusion as part of the FDAP stack: Apache Flight, DataFusion, Arrow, and Parquet.[11] Other users described in public sources include EDB Postgres AI,[12] Cube,[13] Spice AI,[14] Pydantic Logfire,[15] and Kamu.[16]

In 2024, CRN included Apache DataFusion in its list of "The 10 Coolest Open-Source Software Tools Of 2024".[17]

Language support

DataFusion itself is written in Rust. The project also has official Python bindings and community-maintained bindings and tooling for other languages and runtimes.[18][19]

Ecosystem projects

Several projects in the broader Apache ecosystem and the community-maintained datafusion-contrib organization extend DataFusion's capabilities.[19]

References

  1. ↑ 1.0 1.1 1.2 1.3 1.4 1.5 Lamb, Andrew; Shen, Yijie; Heres, Daniel; Chakraborty, Jayjeet; Kabak, Mehmet Ozan; Hsieh, Liang-Chi; Sun, Chao (2024). "Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine". Proceedings of the 2024 International Conference on Management of Data. doi:10.1145/3626246.3653368. 
  2. ↑ 2.0 2.1 2.2 2.3 "Introduction". Apache Software Foundation. https://datafusion.apache.org/user-guide/introduction.html. 
  3. ↑ 3.0 3.1 "DataFusion: A Rust-native Query Engine for Apache Arrow". Apache Software Foundation. 2019-02-04. https://datafusion.apache.org/blog/2019/02/04/datafusion-donation/. 
  4. ↑ 4.0 4.1 "Apache Software Foundation Announces New Top-Level Project Apache DataFusion". Apache Software Foundation. 2024-06-11. https://news.apache.org/foundation/entry/apache-software-foundation-announces-new-top-level-project-apache-datafusion. 
  5. ↑ "datafusion". https://crates.io/crates/datafusion. 
  6. ↑ "SIGMOD 2024 Industrial Track: Accepted Papers". https://2024.sigmod.org/industrial-list.shtml. 
  7. ↑ Pedreira, Pedro; Erling, Orri; Mühleisen, Hannes; Muñoz, Ruben; Khaled, Wael; Dürsch, Peter (2023). "The Composable Data Management System Manifesto". Proceedings of the VLDB Endowment 16 (10). doi:10.14778/3603581.3603604. 
  8. ↑ "Cloudflare Log Explorer is now GA, providing native observability and forensics". Cloudflare. 2025-06-18. https://blog.cloudflare.com/logexplorer-ga/. 
  9. ↑ "Announcements: July 2025". Palantir Technologies. 2025-07-29. https://www.palantir.com/docs/foundry/announcements/2025-07. 
  10. ↑ "Announcements: February 2024". Palantir Technologies. February 2024. https://www.palantir.com/docs/foundry/announcements/2024-02. 
  11. ↑ "Flight, DataFusion, Arrow, and Parquet: Using the FDAP Architecture to build InfluxDB 3.0". 2023-10-25. https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/. 
  12. ↑ "Enterprise DB begins rolling AI features into PostgreSQL". 2024-05-23. https://siliconangle.com/2024/05/23/enterprise-db-begins-rolling-ai-features-postgresql/. 
  13. ↑ "Query pushdown in Cube's semantic layer". 2024-06-03. https://cube.dev/blog/query-push-down-in-cubes-semantic-layer. 
  14. ↑ "How we use Apache DataFusion at Spice AI". 2026-01-17. https://spice.ai/blog/how-we-use-apache-datafusion-at-spice-ai. 
  15. ↑ "We're changing database". 2024-08-29. https://github.com/pydantic/logfire/issues/408. 
  16. ↑ "100X faster ingestion, and FlightSQL support for connecting BI tools". 2023-09-26. https://www.kamu.dev/blog/2023-09-datafusion-flightsql/. 
  17. ↑ "The 10 Coolest Open-Source Software Tools Of 2024". 2024-11-21. https://www.crn.com/news/software/2024/the-10-coolest-open-source-software-tools-of-2024?page=3. 
  18. ↑ 19.0 19.1 "datafusion-contrib". https://github.com/datafusion-contrib. 



Retrieved from "https://handwiki.org/wiki/index.php?title=Software:Apache_DataFusion&oldid=4575104"