IntroducingDuneSQL

TLDR:

• We are investing heavily in DuneSQL to make the fastest, most powerful and convenient crypto data querying experience on the planet.
• DuneSQL will eventually be the only query engine on Dune.
• We have shipped a tool to help Wizards migrate their queries to DuneSQL. We’re committed to helping make the transition to DuneSQL as smooth as possible for Wizards.
• We acknowledge that we have not communicated enough and well enough to the community about the new query engine. We are sorry about that and promise to do better from now on.
• Using other queries as views
• Querying byte arrays directly using 0x-encoding: `WHERE from=0xd8dA6BF26964aF9D7eEd9e03E53415D37aA96045` (notice the lack of quoting or lowercasing)
• Full wei-level precision calculations via UINT256 and INT256 data types.
• ~30% speedup over the alpha launch configuration through better data types.
• Using other queries as materialized views to unlock incredible speedups and new use cases
• Enabling the wizard community to contribute user-defined functions to Dune SQL.
• Query your own data sources directly from Dune.
• Upload data to Dune SQL
• Ability to select performance level for your query
• Dune V1 is in the middle of being phased out as Postgres is not a suitable system for the scale and complexity of our data and pipelines
• Support for everything besides Ethereum on V1 has been deprecated already and only supports editing queries. Many of these datasets are so big that our v1 data platform is struggling to fully support them.
• We’re working with teams to make sure that they can use V2, and we’re very happy to have a dialogue with you.
• Spark SQL will be around for another 3-5 months. There’s no immediate action needed from your side if you are using Dune Engine V2 (Spark SQL), but we highly recommend not creating further queries on it.
• There will be prompts for you to migrate to Dune SQL. Again, Dune SQL has the same data, minus some spells that we are actively working to make available as soon as possible.
• Dune SQL is now the default experience. We are devoting a lot of resources to improve it.
• Start using Dune SQL! If it’s not possible for you to use Dune SQL, reach out to a Dune team member to tell us why
• You can still use Spark SQL for 3-5 months, but Dune SQL is the preferred query engine and eventually you will have to use it.
• Spark SQL on dune.com is not ANSI compliant, which in effect means that it implicitly converts a lot of types for you. When using Dune SQL you may have to explicitly cast types for them to be comparable.
• Spark SQL and Trino SQL (which is what Dune SQL is built on) supports a lot of the same functionality, but their functions and keywords are a little bit different. E.g. `array_contains()` in Spark is just `contains()` in Trino.
• In our experience, the changes required to migrate an existing Spark query to Trino SQL should be relatively straightforward. And because Trino is generally faster for many of Dune’s queries, it should be faster to iterate.
• Some types are different on Dune SQL than on Spark SQL. See here ‍
• For the time being, Spellbook will continue running on Spark SQL. Our Spellbook CI/CD pipeline will validate all new spells or changes are compatible with Dune SQL
• We expect to migrate Spellbook to run Dune SQL in the next 3 months.

‍ ‍

As you come across issues, or areas of improvement, shoot us an email at [email protected] and we will gladly take a look.

‍