👩‍🏭 Work in Progress 🚜

a rant about dbt

November 26, 2023

I've been working with dbt for well over two years now. And I sort of hate it.

I know, I know. There are no better alternatives out there (or are there?). But that doesn't make dbt less terrible.

reasons

dbt is slow

The dbt project I work on is a small one. We have around 100 nodes. Every time I run dbt compile, I have to wait 6 seconds for it to complete. 6 full seconds! This is crazy.

For comparison, our backend is a much larger project. It' not 🍎 to 🍎, I know. But it gives some perspective. It's written in Go. And it takes only 9 seconds to compile. And the compilation done by a proper programming language like Go is much more involved. It guarantees that your program is correct in many ways.

dbt, on the other hand, doesn't even enusre your SQL is syntactically correct.

dbt doesn't understand you nor does understand your SQL

string templating is a mistake

an alternative world

Every so often, I catch my self thinking about writing a proper dbt. I could truly learn about about parsers, lexers, abstract syntax trees. I could write a tool that deeply understands my SQL. Wouldn't it be great to have a blazingly fast™ language server to put autocomplete on the tip of your fingers? A language server to flag a typo in your table name? A language server to let you refactor a your alias TODO on a whim?

Just imagine:

SELECT
    super_very_massive_alias.col1,
    super_very_massive_alias.col1
FROM table AS super_very_massive_alias

~You click a button~ You hit your shortcut -- we're not savages to use our mice after all -- and bam!

SELECT
    i.col1,
    i.col1
FROM table AS i

You have your refactor done automatically. I will write that tool, I think. I will learn rust 🦀 in the process. It will be fast and glorious. But then I remember I have three kids. The boys do need a fatherly presence in their lives. And I go on to run dbt run -s my_def_broken_model and wait 6 seconds for Google BigQuery to tell me through its message-boy dbt that I left a dangling comma after my last CTE.

by way of conclusion

If you have read this far, you already know it: this is a cry for help. So if you know of a better alternative to dbt, feel free to tell me. Here's where to find me: denispmaciel at gmail.com. Halp!