DBT and Snowflake Native app

DBT and Snowflake Native app, how dbt works as a full functional Snowflake native app.

Akhil Gurrapu dbt Snowflake Data Analyst Data Engineering

Mar 11, 2024

Key Points

It seems likely that dbt Cloud is integrated into Snowflake as a native app using Snowpark Container Services, allowing it to run within Snowflake’s environment.
Research suggests the native app includes a custom UI, likely a web application, enabling access to features like dbt Explorer and Ask dbt directly from Snowflake’s interface.
The evidence leans toward the app communicating with dbt Cloud’s APIs to fetch data, ensuring seamless interaction without leaving Snowflake.

Overview

dbt Cloud’s integration as a Snowflake native app leverages advanced container technology to bring its features into Snowflake’s ecosystem, making data management smoother for users. Here’s a breakdown of how it works and its architecture.

How It Works

dbt Cloud is brought to Snowflake as a native app by using Snowpark Container Services, a feature that lets developers run containerized applications within Snowflake. This means dbt can operate its interface and backend directly inside Snowflake, letting users access tools like dbt Explorer and Ask dbt without switching platforms. The app connects to dbt Cloud’s APIs to pull in necessary data, ensuring everything stays secure and efficient within Snowflake.

Architecture

The architecture involves three main parts:

dbt Cloud: The main platform where projects and jobs are managed.
Snowflake Native App: Runs in the user’s Snowflake account using Snowpark Container Services, acting as a bridge to dbt Cloud’s features.
UI Integration: The app likely includes a web-based UI, served by the container, and integrated into Snowflake’s interface for easy access.

This setup ensures users can interact with dbt tools seamlessly, enhancing productivity and data insights.

Survey Note: Detailed Analysis of dbt Cloud as a Snowflake Native App

This note provides a comprehensive examination of how dbt Cloud is integrated into Snowflake as a native app, focusing on the technical architecture and implementation details. The integration leverages Snowflake’s advanced features, particularly Snowpark Container Services, to extend dbt Cloud’s capabilities directly within the Snowflake user interface, enhancing data management and analytics workflows.

Background and Context

dbt (Data Build Tool) is a popular analytics engineering tool used for transforming data in data warehouses like Snowflake. dbt Cloud offers a hosted version with a web-based interface for managing projects, running jobs, and viewing documentation and lineage. Snowflake, a cloud-native data platform, introduced native apps to allow applications to run within its environment, leveraging its compute and storage capabilities. The dbt Snowflake Native App, launched in June 2024 as per Introducing dbt for Snowflake: A Native App launch | dbt Labs, integrates dbt Cloud’s discovery and semantic capabilities into Snowflake, accessible via a single Snowflake login.

Integration Mechanism

The integration is facilitated by the Snowflake Native App Framework and Snowpark Container Services, which became publicly available by December 2023 according to Snowpark Container Services | Snowflake Documentation. Snowpark Container Services is a fully managed container offering that allows deploying, managing, and scaling containerized applications within Snowflake, ensuring data remains within the ecosystem without movement. This is crucial for maintaining security and governance, as highlighted in Intro to Snowpark Container Services, which notes it supports running front-end web applications natively within Snowflake.

For dbt, this means packaging parts of their application, including the UI and backend logic, into containers and running them within Snowflake. The native app, installed in the consumer’s Snowflake account, makes calls to dbt Cloud’s APIs and Datadog APIs for logging, using Snowflake’s external network access, as detailed in the documentation at About the dbt Snowflake Native App | dbt Developer Hub. This setup ensures the app can fetch metadata, execute queries, and provide features like orchestration observability and the Ask dbt chatbot, powered by dbt Semantic Layer and Snowflake Cortex AI.

Architectural Components

The architecture, as inferred from available documentation, includes the following components and interactions, summarized in the table below for clarity:

Component	Description
Consumer’s Snowflake Account	Location where the Native App is installed, powered by Snowpark Container Services. Makes calls to dbt Cloud APIs and Datadog APIs using external network access.
dbt Product Snowflake Account	Hosts and distributes the Native App application package, sharing the consumer’s event table for monitoring and logging.
Consumer’s dbt Cloud Account	Interacts with dbt Cloud APIs for metadata and Semantic Layer queries, utilizing the warehouse for dbt query execution and Cortex LLM Arctic for the Ask dbt chatbot.

This architecture ensures a seamless extension of dbt Cloud’s experience into Snowflake, enabling BI analysts and technical stakeholders to access downstream features without leaving the Snowflake environment.

UI Implementation and Full Stack Functionality

The user’s concern about UI limitations, particularly the assumption that only Streamlit can be used for UI in Snowflake native apps, is addressed by the flexibility of Snowpark Container Services. While Streamlit is commonly used, as noted in Snowflake Native Apps: Our Development Workflow - InterWorks, it’s not the only option. Snowpark Container Services allow running any containerized web application, and for dbt, the UI (e.g., dbt Explorer) is likely a custom web application served by the container. This is supported by Developing Snowflake Native Apps with Snowpark Container Services: Part I | Snowflake Builders Blog, which discusses creating sophisticated web experiences within native apps.

The containerized application includes a web server that serves the UI, and Snowflake integrates this into its interface, possibly through a webview or iframe, allowing users to interact with dbt features directly. This full stack functionality is achieved by running both the frontend and backend within the container, communicating with dbt Cloud for data and leveraging Snowflake’s compute for query execution.

Setup and Prerequisites

Setting up the dbt Snowflake Native App requires specific configurations, as outlined in Set up the dbt Snowflake Native App | dbt Developer Hub. Prerequisites include an Enterprise dbt Cloud account in an AWS or Azure region, a dbt Cloud project with Semantic Layer configured, and ACCOUNTADMIN access in Snowflake, with support for Native App/SPCS integration. The process involves configuring Ask dbt, Snowflake, and dbt Cloud, purchasing and installing the app from the Snowflake Marketplace, and onboarding users, ensuring a smooth integration process.

Implications and Benefits

This integration empowers users closer to the business, as stated in dbt Labs Launches dbt for Snowflake, a New Native App on Snowflake Marketplace | dbt Labs, by providing access to orchestration observability, dbt Explorer, and the Ask dbt chatbot. It reduces the cost of producing insights by centralizing data interactions within Snowflake, enhancing efficiency and trust in data processes, especially in AI-driven environments.

Conclusion

The dbt Snowflake Native App’s architecture leverages Snowpark Container Services to run a containerized version of dbt’s UI and backend within Snowflake, communicating with dbt Cloud’s APIs for data and functionality. This enables a seamless, integrated experience, addressing the user’s query about how dbt Cloud can run as a native app and clarifying that while Streamlit is common, custom web applications are also viable, ensuring full stack functionality within Snowflake.