DuneSQL is our new engine for querying Dune’s V2 data in a faster, more powerful and user-friendly way.
TLDR:
We first announced Dune SQL at DuneCon in Berlin, September 2022. Dune SQL is a query engine and SQL flavor tailored to query experience on Dune. After having launched it in alpha in December, it quickly became the most popular option for querying on Dune, and this week we’re incredibly proud to promote Dune SQL into the default querying experience on Dune. A proper announcement will follow, but I wanted to talk through the changes and the direction Dune is taking, to give you all context.
Writing and executing SQL is at the core of the user experience at Dune.We have a deep understanding of blockchain datasets and the queries the community runs on top. Instead of putting crypto data on off the shelf enterprise query providers we can provide a better experience by investing in our own query engine. Running and modifying not just the infrastructure, but also the software that runs queries means that we can innovate and deliver value faster.
Dune SQL is mostly based on Trino, an open source query engine, and accesses exactly the same data as the Spark SQL engine. Ergonomically, querying in Dune SQL is much like querying in Spark SQL, the main difference being that some functions have different names! More details in our docs.
Dune SQL features many Dune specific enhancements and improvements. Since launching in Alpha in December we’ve enabled
We’re also incredibly excited about our immediate roadmap for Dune SQL:
Dune SQL has been incredibly well received by our community of wizards, and we’re excited to work with everyone to make it even better. However, rolling out Spark SQL and now Dune SQL has admittedly caused some confusion and frustration. We’ve not done a great job at communicating the long term direction, and that’s on me. Expect more and clearer communication from us going forward.
By now, you are probably aware that our Postgres based v1 platform is almost fully sunset, with only the Ethereum dataset still available. This dataset will sunset by July 2023. I’ve included way more context for why Postgres is being sunset further down in the post, if you are interested in the details.
Today, you have two choices when it comes to querying the growing list of datasets on our v2 data platform: Spark SQL and Dune SQL. Going forward, all of Dune’s efforts will be put into Dune SQL, and we are also aiming to sunset Spark SQL by July. This is because Dune SQL is the fastest and most ergonomic engine that we believe will completely change how you work with blockchain data.
While there are syntax changes needed to make Postgres and Spark SQL queries run on Dune SQL, you will find that Dune SQL is faster, and has better ergonomics. In Dune SQL, you no longer need to copy common logic around as CTEs, as you can refer to common logic in other queries using our Queries as Views functionality.
That said, migrating queries from one engine to another while learning a slightly new syntax can be painful, and we know this. This is why we have built a tool to automatically help make query changes for you. This is now available in the query editor with some limited functionality, and we’ll iterate on making it better. The tool will be open sourced, so that wizards can help contribute fixes that in turn help other wizards.
An automated tool will likely not be able to fully translate all queries, but we aim to take away the majority of the complexities you face when moving over to Dune SQL. Our team will also be standing by to help users who are having issues moving over.
Here’s a table to summarize the status
* we are aiming to make Spellbook Dune SQL native before Spark SQL is sunset.
Q: I am using V1, but support for my dataset is disappearing:
I am using v2 (Spark SQL), and I am confused:
What are the main differences between Spark SQL and Dune SQL:
Which engine and SQL dialect should I use for Spellbook contributions?
As you come across issues, or areas of improvement, shoot us an email at dunesql-feedback@dune.com and we will gladly take a look.
The revolution will not be reported quarterly.
Celebrating Dune wizards. Dashboards, stories, alpha dropped in your inbox weekly.