1. Overview (Project Background)

This project was initiated to leverage Databricks as a Lakehouse, the core of the data platform.
The main objective was to migrate data previously operated on BigQuery to Databricks Catalog and create mart tables based on this to enhance the analysis environment.

In terms of BI utilization, the goal was to migrate existing Redash and Tableau dashboards to Databricks Dashboard to build an environment where data analysis and visualization can be operated integrally on a single platform.

2. Solution (Resolution Approach)

This PoC was conducted over 3 weeks, migrating Redash and Tableau dashboards operated on BigQuery to Databricks to establish a new BI environment.

First, we reviewed the existing data structure and access permissions, and analyzed the components and query status of Redash and Tableau. Subsequently, we established data loading guidelines connecting RDS → S3 → Databricks, and restructured the Tableau dashboard on a query-by-query basis.

In the Redash environment, we analyzed and performed the process of converting BigQuery SQL to Databricks SQL, and based on this, implemented Databricks Dashboards corresponding to Redash and Tableau dashboards respectively.
We also conducted functional validation and response speed testing for the Databricks BI environment compared to AS-IS snapshots.

Additionally, we performed testing by integrating Databricks Genie, and utilized the Photon query engine optimized for Apache Spark during the process of converting BigQuery SQL to Databricks SQL.
Users were configured to create and query database objects through Spark SQL syntax.

In the SQL editor, we enabled users to view various result sets and add visualizations using the result panel UI, and in the Redash environment, provided inline help and suggestions through assistant functionality when writing queries.
When using Genie, we validated usability by reviewing responses to each question and, when necessary, training the system to provide correct answers through conversation sessions.

3. Result (Achievements)

As a result of migrating the BI environment to Databricks, dashboard loading speed improved overall.
Based on identical analysis queries, Databricks showed faster response speeds compared to BigQuery connection methods, and during the transition from External Catalog to Standard Catalog and Managed Table, both storage and query structures were optimized together. As a result, stable performance was confirmed even in large-scale data environments.

Additionally, by leveraging Databricks Genie (AI Assistant), we created an environment where business users without SQL knowledge can query data based on natural language. Users were able to quickly derive highly reliable insights without writing code directly, and this significantly lowered the barrier to entry for data utilization.

Dashboards previously operated in Redash and Tableau were also able to migrate to the Databricks environment relatively smoothly.
SQL conversion and visualization mapping proceeded seamlessly, and existing users were able to receive the same or enhanced analysis experience within Databricks without significant learning burden.