SQL Transformation

dbt

When clients need open-source portability or are already on a dbt workflow, we bring the same rigor: tested models, version control, and documentation as code.

What It Is

dbt (data build tool) is the open-source standard for SQL-based data transformation. It compiles templated SQL into warehouse queries, manages dependencies, runs tests, and generates documentation automatically. Works with BigQuery, Snowflake, Redshift, and others. Recent releases add dbt Copilot (AI SQL assistance, GA on paid plans), the Fusion engine (Rust-based, nearing GA, already the default for new dbt Cloud projects), and a GA Semantic Layer powered by MetricFlow.

The dbt Semantic Layer is becoming the foundation for AI accuracy in the data stack. The official MCP server lets AI tools (Claude, Cursor) query MetricFlow definitions directly. The LLM never writes raw SQL; MetricFlow compiles governed queries from metric definitions.

Why We Chose It

dbt is the most widely adopted transformation tool in modern data engineering. When clients have existing dbt projects, multi-cloud requirements, or teams with established dbt expertise, we work within that ecosystem rather than pushing a migration. dbt Cloud also offers a fully managed workflow that mirrors what Dataform provides natively on GCP.

One caveat worth noting: Fivetran and dbt Labs announced a merger in October 2025 (pending regulatory close, expected 2026). That consolidation is convenient for clients on the Fivetran + dbt path, but it's worth staying aware of vendor-lock-in risk. Pairing Airbyte with Dataform (or dbt Core) keeps the ingestion and transformation layers in separate hands.

How We Use It

Build and restructure dbt projects following the staging → intermediate → marts layer convention

Write Jinja-templated SQL macros for reusable transformation logic

Implement dbt tests (not null, unique, referential integrity, custom) across critical models

Configure dbt documentation and lineage graphs so new team members understand data flow

Set up dbt Cloud jobs with CI/CD integration: run tests on PRs before merging

Connect the dbt MCP server to the client's AI tools (Claude, Cursor) for natural-language queries over governed metrics

dbt or Dataform?

Choose dbt when:

  • You have existing dbt investment (projects, CI/CD, team knowledge)
  • You need multi-cloud portability (BigQuery + Snowflake, or moving later)
  • Your team already works in dbt every day

Choose dbt Cloud when:

  • You want managed CI/CD, documentation hosting, and the Metrics Layer without DIY
  • You're comfortable with the price (starts around $100/developer/month)

Choose dbt Core when:

  • Your budget rules out dbt Cloud
  • You're willing to run orchestration yourself (usually Dagster or GitHub Actions)

Choose Dataform instead when:

  • You're on GCP only, with no multi-cloud plans
  • Your team is new to analytics engineering
  • You want to skip the orchestration layer entirely

No data platform yet?

Build end-to-end from scratch — ingestion, transformations, dashboards. Audit first, then a fixed-price build.

See Implementation

Platform exists but not working?

Audit what you have, then build the improvements. Two phases — audit first, then a fixed-price build.

See Optimization

Need help with dbt?

Focused help on one or two layers — Dataform, Looker, BigQuery costs. Fixed-price, 2–6 weeks.

See Expert Services