My problem is the opposite. In engineering, the silent killer isn't the inability to start, it's the inability to stop. Call it hyperfocus, call it perseveration: when a problem catches, everything else drops. Other work, sleep, conversations. Gone. I don't resurface until either the thing is done, or I am.

I've learned to build guardrails. Social media was the easy one: an algorithmic feed is designed to keep you consuming, never surfacing. No exit by design. That loop has nothing worth losing sleep over, so I mostly stay away.

But work is different. The problem isn't going to solve itself, so I have to dive in. I've tried hard rules, like no work after dinner, to prevent those sleepless nights. But rules only hold until you're three errors deep at 11PM and the fix feels right there.

Here's the thing I kept sacrificing to stay functional: exploration. Not mindless scrolling, but actual cognitive wandering, letting attention drift sideways, follow tangents, consider the adjacent possibilities. That's where most of the interesting things come from: unexpected connections, better questions or simply the solutions you can only reach by first taking a step back. But wandering only works if you don't get lost in it. For me, that was never guaranteed. Once something caught, I tended to follow it all the way down. So I learned to keep a tighter leash on my own curiosity, just to get work done. Guardrails on everything, including the good stuff.

But that has changed with AI. Not that I suddenly became disciplined. But AI has made a lot of those descents shallower, and that has quietly given me something back...

The old research loop used to be: query, ten tabs, three more queries, another page of results, compare sources, slowly build a mental model. Now it's much more often query, read the synthesis, then sources only to dig deeper where it actually matters. And for the technical rabbit holes, the effect is different but just as important. When I hit a gnarly implementation detail, a weird dependency issue, or some cursed edge case, I don't always have to go all the way down myself anymore. A coding agent can take care of that for me.

Which means my attention is freer than it used to be. Not because I suddenly stopped going deep, but because depth no longer swallows me whole.

I'm still figuring out what that actually means for me. But in the middle of last night, attention spinning in the dark, I landed on this: attention isn't binary. It's a dial. You can turn it up for depth, down for breadth. But the part that used to feel impossible: getting my attention back when I need to move on. That’s what actually feels new.

Now, if you'll excuse me, it's time to turn the dial down and step AFK.

PS: the AI might be solving my daytime rabbit holes but at night, trying to sleep, I'm still left alone with my meaty brain, battling random 3AM inspiration spikes! That's rarely enjoyable: tomorrow-me always pays the price. If you've found a way to quiet your brain for the entire night, every night, and without the grogginess the next day, I'm genuinely all ears.

If a "simple" KPI like Sales by Region keeps changing every time someone moves house or changes team, that's not your dashboard being flaky. That's your data model telling you it only remembers the latest truth.

As long as dimension tables keep one row per customer/account/contract, they can only answer one question:

"What does this entity look like right now?"

The questions stakeholders actually ask are different:

• "What did this customer look like when they ordered?"

• "Who owned this account when the deal closed?"

• "What was the contract status at quarter-end?"

To answer those, the warehouse needs some form of time travel.

In an ideal world, that comes from CDC (Change Data Capture): reading the database's transaction logs (WAL, binlog, etc.) and replaying every insert, update, and delete as a continuous event stream.

In the real world, that often means begging infra for Debezium/Kafka/cloud‑$VENDOR$ you will never get.

This is where dbt snapshots are useful: a very simple, very boring way to add just enough history to make your warehouse behave like it remembers the past.

1. The real problem SCD Type 2 solves

TL;DR: Without SCD Type 2, the warehouse only remembers the latest state. With it, you can reconstruct what things looked like at any past date.

Think about a classic e‑commerce case: "Sales by Region" over the last year.

• A customer lives in New York for 6 months and buys 5 items.

• Then they move to California and buy 3 more.

If the customers dimension only keeps their current address:

• All 8 orders get attributed to California.

• New York's historical revenue is under‑reported.

• California looks more successful than it really was.

The warehouse is effectively rewiring the past every time someone moves.

SCD Type 2 is the pattern that fixes this. It lets the dimensional model answer:

"What was the state of this customer at the time of each order?"

Once that exists, a whole class of questions becomes actually answerable:

• Sales attribution by region / rep at deal time

• Funnel conversion by segment at the time of signup

• Regulatory views of contracts as of reporting date

Without SCD2, those are guesswork. With it, they're just SQL.

2. SCD Type 2 in two columns

TL;DR: Add valid_from and valid_to to the dimension, and stop overwriting rows.

Instead of overwriting a customer row when something changes, SCD Type 2 adds a new row and marks the old one as expired.

Minimal version of a customer address history table:

customer_id | name | region | valid_from  | valid_to
----------- | ---- | ------ | ----------- | ----------
CUST-001    | John | NY     | 2024-01-01  | 2024-07-01
CUST-001    | John | CA     | 2024-07-01  | NULL

• customer_id: same across all versions for the same person.

• valid_from: when this version became active.

• valid_to: when this version stopped being true. NULL = "still active".

dbt will also add a surrogate key like dbt_scd_id under the hood, one per version, which is what you'd typically join from facts in a more "by‑the‑book" warehouse.

Point‑in‑time lookups

To know where John lived on March 15th:

SELECT *
FROM customer_scd2
WHERE '2024-03-15' BETWEEN valid_from AND COALESCE(valid_to, CURRENT_DATE);

March 15th sits inside the [2024‑01‑01, 2024‑07‑01) window, so you get the NY row.

That one pattern—"date BETWEEN valid_from AND valid_to (or COALESCE)"—is basically the entire mechanic.

3. dbt snapshots: an easy batch SCD2

TL;DR: dbt snapshots implement SCD Type 2 for you. You point them at a table or model, configure how to detect changes, and run them on a schedule.

What snapshots actually do

On each dbt snapshot run, dbt:

1. Reads the current state of your source/model.

2. Compares it to the existing snapshot table.

3. For rows that changed:

   • Finds the existing "current" version (where dbt_valid_to IS NULL) for that entity.

   • Updates it by setting dbt_valid_to = snapshot timestamp.

   • Inserts a new version with dbt_valid_from = snapshot timestamp, dbt_valid_to = NULL.

4. Leaves unchanged rows alone.

5. Keeps current rows with dbt_valid_to = NULL (or a configurable max date like 9999‑12‑31).

You get:

• dbt_valid_from

• dbt_valid_to

• dbt_scd_id (surrogate key, unique per version)

• Optional extras (like dbt_is_deleted if you enable hard deletes).

How to configure snapshots (modern way)

As of dbt Core v1.9+, snapshots are configured in YAML, not in .sql blocks. If any transformation is needed, do it in a model (ephemeral or not), then snapshot that.

Example: snapshotting a raw customers table:

# snapshots/properties.yml
snapshots:
  - name: customer_snapshot
    relation: source('raw_database', 'customers')
    config:
      schema: snapshots
      unique_key: customer_id
      strategy: timestamp
      updated_at: updated_at

Key bits:

• relation: what to snapshot (source() or ref()).

• unique_key: stable business identifier for the entity (identifies the "same customer across versions").

• strategy: how to detect changes.

• updated_at: which column to use for timestamp strategy.

• schema: where to store the snapshot table.

Optional, but very useful:

• dbt_valid_to_current: store a max date instead of NULL for current rows (e.g. to_date('9999-12-31')), so queries can use a plain BETWEEN without COALESCE.

• hard_deletes: 'new_record': add rows for hard deletions with a dbt_is_deleted flag, instead of silently losing them.

Two change‑detection strategies

dbt supports two built‑in strategies to decide when "this row is different enough to version it."

1. timestamp (use this whenever you can)

strategy: timestamp
updated_at: updated_at

• Compares only the updated_at column.

• If updated_at is newer than what's in the snapshot, dbt records a new version.

• Handles schema changes gracefully—new columns don't require config changes.

Constraint: updated_at must actually change whenever any tracked attribute changes. If the source only bumps it for some columns, you will miss changes in the others.

2. check (fallback when no good timestamp)

strategy: check
check_cols:
  - region
  - customer_segment

• Compares the actual values of the listed columns against the snapshot.

• If any differ, new version.

Pros:

• Works when there's no reliable updated_at.

• Explicit about which attributes you care about.

Cons:

• Scales worse than timestamp as rows get wider and tables bigger, because more data must be compared per row.

• Requires config maintenance when adding/removing columns.

Good rule of thumb:

If there is a trustworthy updated_at, use timestamp.
If there isn't, use check on a small, intentional subset of columns.

How often to run snapshots

dbt's own recommendation: somewhere between hourly and daily. If you feel tempted to run snapshots every minute, you're trying to force a batch tool to behave like real‑time CDC.

• Run too rarely → you miss more behaviour between runs.

• Run too often → you're paying a lot for a poor approximation of a log‑based system.

Running snapshots vs `dbt run`

Snapshots are not run as part of dbt run.

• dbt run materializes your models.

• dbt snapshot runs only your snapshots.

So you’ll generally want to schedule both:

• dbt snapshot on your chosen cadence (hourly/daily), and

• dbt run for your transformation models.

Note: Snapshots only track changes from the moment you create and start running them. They can't reconstruct history from before the first snapshot run.

4. Putting it together in SQL

TL;DR: The pattern is always "join facts to the snapshot where the fact's date is between dbt_valid_from and dbt_valid_to (or its COALESCE)."

Point‑in‑time lookup

Basic "state on a given date":

SELECT 
  customer_id,
  region
FROM customer_snapshot
WHERE 
  '2024-03-15' BETWEEN dbt_valid_from
                   AND COALESCE(dbt_valid_to, CURRENT_DATE);

If you configure dbt_valid_to_current to store 9999‑12‑31 for current rows, the COALESCE can go away:

WHERE '2024-03-15' BETWEEN dbt_valid_from AND dbt_valid_to;

Sales by customer region at order time

SELECT
  cs.region,
  SUM(o.order_total) AS total_revenue
FROM fact_orders AS o
JOIN customer_snapshot AS cs
  ON o.customer_id = cs.customer_id
 AND o.order_date BETWEEN cs.dbt_valid_from
                      AND COALESCE(cs.dbt_valid_to, CURRENT_DATE)
GROUP BY
  cs.region;

The join condition is doing all the temporal heavy lifting: each order row finds the version of the customer record that was valid on the order date.

The result:

• The 5 "New York" orders are counted for NY.

• The 3 "California" orders are counted for CA.

• No one has to hack around it in Looker/Power BI with window functions and CASE statements.

5. When snapshots are enough, and when you really need CDC

TL;DR: Snapshots are batch SCD2. They capture differences between runs, but not everything that happens in between those runs.

What snapshots are genuinely good at

• Capturing most business‑relevant changes in CRUD‑style apps.

• Making point‑in‑time joins trivial.

• Giving analysts a reliable way to time‑travel dimensions using standard SQL.

• Providing a simple audit trail for "what did this row look like before/after X?".

All with:

• No log readers.

• No Kafka, unless you otherwise need it.

• No extra infra dependencies beyond "warehouse + dbt + scheduler".

Built‑in limitations (by design)

Snapshots cannot and will not:

• Show every intermediate state between runs. If a field flips A → B → A between two snapshots, the snapshot will never know it changed at all.

• React in real time. Until the next run, the snapshot is out of date.

• Magically turn a flaky updated_at into a reliable change log. Garbage in, garbage out.

If a regulator, auditor, or core business requirement says:

"Every change to this record must be permanently recorded, in order."

then snapshots are the wrong tool. That's where real CDC—consuming transaction logs or event streams—earns its keep.

A healthy mental model:

• Snapshots = batch time‑travel for analytics.

• CDC = continuous ledger for systems that must never forget anything.

“But why not just freeze the numbers?”

You can also solve some of these problems with incremental data marts:

• Each run upserts a new batch of results for the current period (for example, sales_by_region_2024_Q1).

• Older batches are never updated, so your dashboards don’t change after the period closes.

This works, but there’s a trade-off:

• If your business logic changes later (new definition of “active customer”, new region mapping, etc.), you simply can’t re-run the past without rebuilding the whole mart.

• So you’d either live with inconsistent logic across periods, or you’d lose the original “frozen” numbers.

Snapshots give you a different superpower: you preserve row-level history and can re-run models with new logic while still joining facts to the state that was true at the time.

6. Operations: keeping history without drowning in it

Snapshots accumulate forever. That's their job. But large history tables can still hurt if left completely unmanaged.

Practical guardrails:

• Partition by dbt_valid_from (or similar) so most queries only scan relevant periods.

• Cluster on a key like customer_id or order_id to speed up point‑in‑time joins.

• Retention rules: if no one cares about row‑level history older than a few years, consider archiving or aggregating old partitions.

None of this is specific to dbt; it's just basic warehouse hygiene applied to a table that, by definition, only grows.

Summary

Next steps

Set up a small snapshot on a single dimension (like customers), wire one existing report to use it, and watch how many "wait, why did this change?" conversations disappear. Once the pattern clicks, the rest of the team can use it everywhere without having to understand CDC or event logs.

Note: This article builds on the official dbt snapshots documentation and config reference changed with dbt Core 1.9 and documented on the dbt Developer Hub as of January 2026 (current dbt Core version at time of writing: 1.11.2).

Here's the gist of how that went:

Change data capture (CDC) is basically reading the event stream of everything that has happened on a database. Any update, insert, delete (and sometimes even DDL commands) that have been ran on the database. The database itself actually uses that same stream to apply the requested changes/events to the main node, as well as any replicas it might have. CDC therefor is already used internally by the database itself.

That being said, when we talk about CDC as data engineers, more often then not, we're referring to reading that same event stream but for "external" consumption/replication (i.e towards a data warehouse or datalake, generally for post-hoc analytical purposes...)

And that got me thinking that the underlying concept: the event stream, is nebulous at best... Which is a shame, because it's really useful!! So it would be worth being able to explain it to anyone, even my 6yo son. (now 8yo 🤦)

Anyhow, let's get on with the story, shall we?

The Never-Ending Book

Long ago, storytellers could only share tales by speaking out loud. If you wanted to hear their stories, you had to be there listening at that very moment. If you left or got distracted, you'd miss out on parts of the narrative.

But then, writing came along. Suddenly, storytellers could write their stories down in books. This allowed listeners to become readers who could pick up where they left off whenever they wanted. The storytellers no longer had to wait for an audience before sharing their tales.

Now, imagine there was a magical author who was so inspired, her stories never ended! She just kept writing and writing without ever stopping. To share this endless tale, she created a very special magical book.

This book was truly magical because no matter how many new pages the author added to it, every reader could see the full book at the same time! The author could keep writing forever, with new pages just appearing at the very end for all to see.

Since readers all experienced the story at their own pace, each one needed a special bookmark to keep their place. Thanks to the bookmarks, they could put the book down and come back to it later without losing their spot.

The author's inspiration was so endless that the book just kept growing bigger and bigger and bigger. Before long, it became too huge for anyone to pick up!

So the author cast another spell: This time, as soon as every single reader had finished with a page, that page would magically disappear from the front of the book. This kept the book from becoming too heavy to lift.

But what if a reader couldn't read for too long, or stopped altogether? If their bookmark didn't move for a while, it would eventually disappear too so it didn't slow down all the other readers.

And that, kids, is a lot like the "event stream" systems that power so many of the apps and websites we use today...

There you have it, that's basically the same as an event stream. New pages (or events) keep getting added at the end. And pages that have been read by all readers (consumers) can be discarded along the way.

This illustrates:

• the need for asynchronous communication to start with

• how event streams (or message queues) enable such communication

• the never ending aspect of streams

• as well as the basics around keeping it within reasonable size.

However it does not explicit the fact that readers can consume concurrently, which is the magic scaling benefit of such systems (when configured properly). But I find it still captures most of the intuition around such a piece of stack.

And to close off the loop: that's also how the Postgres Write Ahead Log (WAL - used for CDC) generally works and can be configured...

So what's this dbt everyone's fussing about??

At its core, dbt (data build tool) is a "modern" data modeling framework using (Jinja) templated SQL. It comes with a CLI tool that allows to easily materialize your models as tables or views in your data warehouse.

The company behind it: dbt Labs (formerly known as Fishtown Analytics) also produces a cloud-managed service revolving around the core dbt. They call it, quite simply, "dbt Cloud". Now, we won't get into that part here, just a thing to note: they've been central to the current "modern data stack" movement, for example coining new job titles like "analytics engineer" and, doing so, they've coalesced 😉 quite a following.

What I mean (and think most people mean) by "modern data stack" is any service (open-sourced or not) that is native to cloud data warehouses (things like Snowflake, BigQuery and co...) and specifically targeting data teams.

What dbt allows is quite useful for data teams. Simply put, we're now easily able to adopt best practices software engineers take for granted (things like code versioning and reviewing, sandbox environments, testing, documentation, etc.) throughout our data warehouses and data modeling processes.

That makes things like unreproducible queries or inextricably weird numbers things of the past! Because instead of having a collection of interdependent views and/or dynamically generated bits of queries that do stuff all over our warehouse, it gives us (pretty) plain SQL select statements, cleanly versioned in a git code repository.

Along with a simple CLI to run these models, which will "materialize" (create or update) them in our data warehouse. It also has a bunch of nice additions that help us structure, test and document all those data models.

Disclaimer: We'll cover how to gradually structure a dbt project. Not how to run it. When you're developing you can install the dbt CLI to easily use it in your development environment. For deployment to production there are different options. If you're already using a data orchestrator like Airflow, Prefect or Dagster, running dbt is as simple as adding a task/step that runs dbt build in a shell environment containing the dbt pip package you're using.

With great power comes great responsibility

The problem with frameworks that are so flexible is that they can be used in way too many ways. Some are good, some less so...

Anyway, we know we want to use dbt, but how should we go about it?

In terms of data modeling, there are multiple theories out there. Take one of the most prevalent still widely used today, dimensional modeling: it was initially formulated back in the 90s’!

Now don't get me wrong, it has plenty of still very relevant and useful principles. The problem though, is that it takes too much conception time to pay up front. That's because it's mainly top-down design, at least for the initial fleshing out of the business entities and relationships.

In contrast, the approach I want to outline here is primarily bottom-up, in order to:

• deliver value and insights as quickly as possible

• not pile up too much (data modeling) technical debt along the way.

So, we'll start naively, and gradually factorize common patterns as they emerge.

Quick and dirty... or let's say: humble beginnings

The quickest way to extract any insight from our data is to:

1. have all our data sources readily available under the same roof: in our data warehouse (I’ve mentioned a couple options to help with that in a previous post)

2. directly write each query that will join all the useful sources into a single table that we’ll then expose to our stakeholders (either in their own tools via reverse-ETL or through a dashboard we'll create for them in our BI tool of choice)

Of course, this isn’t sustainable! Because as soon as we have piled up a few such usecases , we'll surely need to copy/paste some parts from one query to the next... And this is always a slippery slope to get on!

Still! Go ahead and hack together your first couple of usecases in precisely this manner. The quicker we prove the value of data, the quicker we’ll start building support for “data” as a function/team within our organization. And that’ll be our basis for hiring more, spending more, etc. So raw speed is definitely something we want to aim for... in the beginning.

First off, dbt init your git repository. dbt will create a folder for you (named with the project name you’ll enter at prompt), which will contain the whole dbt project: meaning the .sql queries/files we’ll add, and a few .yml configurations to go with 'em.

That first command will have created a bunch of folders, most of which we won’t get into right now, but the main ones we can start working with are analysis and models.

• analysis can hold all our ad-hoc queries that we'll want to manually run,

• models will contain all the queries that we actually want to dbt run, which will then materialize (as tables or views) in the data warehouse.

Also, it loaded a couple of useless examples in the models folder. You can safely delete these. They only sparsely illustrate dbt’s main Jinja "macro" when you’re starting out: {{ref(<parent_model_of_current_query>) }} but we’ll get into that later 😉

You can simply start creating .sql files with your select queries for your first few usecases. These can go in the models folder if you want them to be materialized on your data warehouse when you execute dbt run from your CLI.

When running dbt commands, it isn’t really doing anything more than just executing our SQL files (just wrapped up in the correct DDLboilerplate) against our warehouse at this stage. We’ll get to the Jinja templating superpowers later!

If in doubt, please check out dbt’s documentation to get you up and running: Connection profiles.

The only useful piece of advice at this stage: do start splitting out the logic within your query/ies in smaller bite-sized CTEs. This will be a huge time saver down the line.

But even with somewhat readable queries, just be aware that you’ll soon hit that slippery slope we mentioned earlier.

Start specifying your sources

So the next step we should take is to add a first base layer in our dbt project. I like calling it "base" as it’ll be the layer in which we simply select the data we need from our raw data sources.

So let’s create that base folder under models/. Then add a general _sources.yml inside. That configuration file will hold all the (input) references to our raw data. Effectively declaring the origin of your data sources (read the dbt documentation linked to get the details of what that means and looks like).

You can now start creating simple select statements for the data sources that are actually used in the usecases you’ve previously fleshed out. The idea here is to be explicit with the columns you’re selecting. Try not to be tempted with just using select * because the main responsibility of this layer will be to explicitly declare the data your using at the column level.

Our _<source_name>_sources.yml declares where our raw data is, our base models declare what raw data we want to use.

Here’s a quick example (in Snowflake syntax):

-- base/mailchimp/base_mailchimp__members.sql

with parsed as (
    select *
      , parse_json(merge_fields) as parsed_fields
    from {{ source('mailchimp', 'members') }}
)
select id as member_id
    , status
    , parsed_fields:'FNAME' || ' ' || parsed_fields:'LNAME' as full_name
    , parsed_fields:'REGISTERAT' as registered_at
from parsed

We can then use this with select * from {{ ref('base_mailchimp__members') }} in a downstream model/query...

Just don’t forget to point your usecases to the new base models created here with that ref() macro.

Now, under this new base folder, we can also separate each data source into it’s own subfolder (as shown above). This allows to cleanly organize our base layer by source. That being said, we do not want to nest too many levels of folders. For the simple reason that dbt lives in a single namespace!

This means all our models (models/**/*.sql files) all need to have a unique name. A good idea is to prefix the filenames with the source’s name (as shown above). This will prevent most duplicate name clashes. Bonus: let's start writing down a style guide where we'll, ultimately, specify our preferences in terms writing SQL, but for now simply document this naming convention.

i.e "base models filepath should be in the form: base/<source_name>/base_<source_name>__<model_name>.sql"

Apart from explicitly detailing what data we want to use from our sources, any simple row-wise transformations can also go here in our base layer. Things like renaming columns to align names across models, type casting or even "case-when-ing" 😉 Here, we just want to avoid changing the grain of our raw data (no group bys nor joins) or filter anything out.

Last but not least, take the time to add a _<source_name>_models.yml file to each of the base subfolders. There, we’ll start specifying descriptions and schema tests for the most used models and their main columns. You don’t need to be exhaustive for the time being. Just describe and test the most important ones to start off with!

Now that we've got our bases covered, where to?

First refactorings

Now it’s time to add a middle layer (still under the models/ folder). You can call it core, staging, transform or anything in between: this is ultimately where most of the magic will happen. The layer that our data team will keep full ownership of till the end of times! (no pressure 😆)

But for now, let’s start simple: is there any logic that you’ve already had to copy/paste in the different usecase queries you already have? Or CTEs you keep on re-writing in each new usecase?

If so, just isolate the repeated parts in their own .sql file in this folder and ref() it in the usecase models that use it.

For the time being, the main point is to start reducing the complexity and length of the usecase models by simply extracting the parts that are repeating. A good rule of thumb is the rule of three: if you have a logic that’s repeated 3 (or more) times in your project, then it should be isolated in it’s own model and can then be referenced by all the models that use it.

This way, if that logic needs to change down the line, it can be changed in a single place, instead of having to patch it systematically in all the places where it’s used. The risk being that it's easy to miss one of these places and therefor have differing logic when it should really be the same.

If your main usecase queries are still directly under the models folder. You can move them to models/usecases/. And, if you’re starting to have more than a few usecases, a nice addition to that folder is a further level of subfolders per stakeholder type. E.g one for usecases/product , another for usecases/sales or usecases/marketing, yet another for usecases/finance or usescases/board, ... You get the idea.

The ownership over each of these can ultimately be split out if and when data competency is spread in each department of our organization. But that’ll be the topic for another blog...😉

Nice: our project is getting a bit more structured!

So we can now start getting into the Jinja templating goodness of dbt. But let's use it wisely 😆

Good candidates for such templating are if you have long and predictable column lists that you want to systematically add. You can do so with a simple Jinja for loop. You can also check out the dbt_utilspackage for other ideas on what you can do more concisely with Jinja.

Any custom Jinja macros should go in the macros main level folder of your dbt project (same level as the models folder). I generally like separating out the macros that simply override dbt core ones in a macros/overrides subfolder. The first example of such, is generally the generate_schema_name.sql macro to specify different schema names between locally/manually triggered runs and non-dev (prod/preprod) environments for example.

Here’s how I often end up doing that:

-- macros/overrides/generate_schema_name.sql

{% macro generate_schema_name(schema_name, node='Unknown node') -%}
    {%- if var('ENV') == 'dev' and target.schema -%}
        {# -- use the target schema provided in profiles.yml for manually-triggered (dev) runs #}
        {{ target.schema | trim }}
    {%- elif var('ENV') in ('prod', 'preprod') -%}
        {# -- or use the schemas provided in configuration (in dbt_project.yml or individual models' config) #}
        {{ schema_name | trim }}
    {%- else -%}
        {# -- fail fast in case of unexpected configurations #}
        {{ exceptions.raise_compiler_error("Local run but environment variable `DBT_DEFAULT_SCHEMA` is not set. Or environment is not known.") }}
    {%- endif -%}
{%- endmacro %}

Which requires adding to the end of your dbt_project.yml:

#...
vars:
  ENV: "{{ env_var('DBT_ENV', 'dev') }}"

And passing a DBT_ENV environment variable with the corresponding environment name.

Next up: shoot for the stars!

Endgame: core objects

Now that we’re getting the hang of it, it’s time to shuffle things around a bit more 😆

That middle layer we’ve introduced is actually the place where we’ll want to have our central logic manipulating the data.

The goal will be to aim towards either:

• a dimensional model/star schema

• or a limited collection of "One Big Tables"

• (or any other top-down theory you’re most comfortable with...)

where we’ll progressively define all our main business objects: users on the one hand, orders on the other, inventory in a third, etc

Each being at the grain of that object and having all the attributes useful to that particular object (via shared dimensions in the case of dimensional modeling).

These core objects will be the output of that layer, but will probably each need to be based on shared "preparation" models (the repeating CTEs we first factorized above...) Which can go in a core/stagingsubfolder when things become unwieldy.

As previously mentioned, we’ll generally want to keep central ownership on that layer.

And from these core objects, the usecases that are exposed (then outsourced) to our different stakeholders can simply select a limited number of columns/informations, relevant to each particular usecase.

So from sources -> base -> core, we're integrating data: the idea being to end up with fewer but wider (more columns) models.

And from core -> usecases, we're doing a fan-out, producing multiple variations from the core objects that only specify the data needed for each different type of stakeholder.

Conclusion: don't over design up-front

"Be like water my friend"

1. Start small but start fast

2. Progressively regroup repetitions

3. Only formalize once you have a good grasp of what's what

4. Rinse and repeat...

Agile analytics engineering.

Addendum: Naming is important

Why do I like these 3 folders for the first level under models?

1. base

2. core this folder we’ll always be our common thread as data modelers, our perpetual “work in progress”

3. usecases these models are our end goals and we want the insights to actually be “useable”! (detail: I like the plural to insist on the transverse role of our data team: we strive to serve all the other teams!!)

Ok, these names can seem a bit arbitrary, because they kind of are. But another thing I really like is that they’ll appear in that logical, sequential, alphabetical order anywhere you’ll list them: your cloud data warehouse UI (cf config in snippet below), your file explorer, your IDE of choice, you name it!

It just flows in the natural order: starting from our base models, passing through our core and finishing with the actual usecases we want to deliver.

Of course, I encourage you to align that first level of folder structure with your warehouse schemas. Here’s how in your dbt_project.yml:

models:
  <your_dbt_project_name>:
    base:
      +schema: "base"
    core:
      +schema: "core"
    usecases:
      +schema: "usecases"

Note: this will only be the case in prod/preprod envs if your using generate_schema_name macro covered previously, when triggering locally (dev) everything will end up in a single schema (defined in your personal ~/.dbt/profiles.yml)

Published by:

Chances are you’ve tried both Metabase and Redash. Both being lightweight and open source alternatives to more full-fledged BI solutions like Superset (also open source), Looker, Mode or Tableau.

But if you’re lucky enough to get some traction internally on your brand new BI tool’s usage, you’ll soon end up with a sprawling collection of disparate queries and dashboards. Because everyone will be trying to find their own answers.

And that’s when things generally get messy… with other additional issues you’ll eventually encounter:

1. Your production database won’t scale to handle the analytical workloads on top of your transactional ones.
2. Your production database won’t contain the data of your whole business, only from the app it’s serving (no data from your CRM, ERP, ad spend, etc)
3. Your production data won’t be ready for easy human consumption, quite the opposite, it should only care about being correctly normalized for your app’s consistent usage.

Now, don’t get me wrong, you can get away with those for some time, by “simply” vertically scaling your replica, by adding other data sources as additional “connections” to your BI tool, and by educating all data users on each and every separate data source available.

Or, you can take the next real steps on your analytics journey: bootstrapping your Modern Data Stack.

With this article, I want to provide some simple — albeit opinionated — next steps you should take to build your data-driven company. The idea is to prevent you from getting lost in too much detail!

Personally, I’ve been into data engineering for a few years now and have had the chance of setting up such stacks for a bunch of startups and scaleups over the past couple of years.

Step 1: set up a cloud data warehouse

“Where they are put in boxes, and they come out all the same.”

In order to decouple the scaling of your app’s transactional workloads from that of your analytics workload, the idea is to add a resource specifically tailored to the latter.

Furthermore, the aim will be to make it your central source of truth for what’s going on across your whole business.

In terms of options, general consensus among data engineers currently has it that you can’t go wrong with either Big Query or Snowflake.

Big Query:

• if you already rely on GCP for other stuff in your tech stack
• or you have some machine learning use cases you already want to teeth out (as BQ is pretty tightly integrated with Google’s ML services, among others)
• and/or you prefer a less hands-on approach to elasticity and scaling

Snowflake:

• if none of the above really matters
• and your team has some — even very limited — experience with database administration: you’ll feel right at home here (e.g user management through DDL statements)

Step 2: extract & load into your warehouse

“Lock and load.”

Now that you have a brand new and immaculate data warehouse, it’s time to start loading things in it.

While we’re at it, why not dump every potentially useful data source from across your business?

Typically, your future rockstar CMO might be interested in measuring your ads’ ROI (i.e Is our spending on Facebook Ads leading new users to sign up? Or maybe bringing past ones back? …)

You have quite a few options in terms of SaaS providers to do precisely that.

What you’ll generally get, are click-based UIs that allow to quickly connect your database(s) and/or a bunch of third party APIs as data sources to be loaded into your warehouse.

This even allows less technical folks to quickly become independent in dumping the data from their favorite tool right into the common warehouse.

Here, I’d highlight Fivetran — current EL(T) market leader — as the more mature option, albeit currently very pricey.

Or Airbyte as the latest but strongest upcoming contender. Especially if you’re comfortable hosting you’re own ingestion infrastructure: it’s open sourced and just a docker-compose up away.

Step 3: model, model, model

Also known as the ‘T’ in ELT (i.e “Transform”)

What you’ll need now is to get usable results — KPIs — from all of that data. Maybe merging data across different sources, aggregating it per user segment, and/or per day, week, month, quarter? Comparing that year over year, or over the previous period?

There are quite a few specialist terms involved because of a long history of techniques used to do that holistically.

You might have heard of things like Kimball, Dimensional modeling, Star schema (all three are often used interchangeably) or others: Datavault, Puppini bridge, etc.

These techniques have roots back in the 80s’ when on-premises proprietary data warehouses were the norm (and ETL was a necessary evil). So the arguments focused on performance are way less relevant nowadays with cloud data warehouses being the norm.

But all the other benefits remain essential.

The problem is, in a start up environment where everything is going too fast, taking 6 months to design that perfect star schema from scratch will rarely be an option.

Hence my general advice in that context is: start from the use case and work your way back to the data sources (not the other way around).

Then you’ll refactor into more generic and intermediate layers/models once multiple — think rule of three — use cases need similar transformation logic applied upstream.

A framework that really helps pinning all of that down is dbt.

How does it help ?

It allows anyone that speaks SQL to contribute to your shared data warehouse construction. All of that in a versioned, tested and templated way, with builtin dependency resolution between all your models and sources.

Then, what’s next?

Give back!

This one should actually go hand in hand with that previous step. Focusing on the actual use case(s) is what will align the data team to quickly bring strong value to the table.

This can take multiple forms depending on the use case at hand:

• from dashboards (or alerts) to allow the whole company to gain insight on the business
• to reverse ETLs that’ll make that transformed data directly actionable by sharing it back to the operational tools the other teams are using (e.g calculate your segments in your data warehouse to then (re)load them in your email marketing tool, weighing clients at risk for your CRM, …)
• or serving data APIs to feed some insights/recommendations/predictions into your product directly

Epilogue

Something I could’ve mentioned: of course you’ll need to start recruiting some data specialists along the way. A data analyst or two for starters, helped by an analytics engineer.

I’ve held back on that one to highlight the fact that: there are quite a few things you can progress on, even before that very first data hire! That would definitely help with the hiring process, by conveying the importance you want to place on having a sustainable data stack. And it could actually enable them to make an impact from the get-go.

There you have it. You’re just a few steps away from data-enabled greatness! So start building ;-)