How I Built a Scalable Marketing Dashboard with Airbyte, BigQuery, and Looker Studio
Hi, I’m Mohamed Shaheen, CEO and co-founder of Appgain. Managing our fintech app alongside our e-commerce platform means my days are packed with strategic decisions—from product innovation to customer engagement. One recurring challenge was keeping a real-time pulse on our marketing performance without incurring excessive costs.
In our early days, we relied on Funnel.io for our marketing dashboards due to its ease of setup. However, as our business expanded and our marketing channels multiplied, the cost of maintaining dashboards grew significantly. We needed a more flexible and cost-effective solution.
After thorough discussions with our in-house data engineers, we decided to build our own marketing dashboard using an open-source stack: Airbyte for data extraction, Google BigQuery for data storage and processing, and Looker Studio for visualization. This is the story of how we transformed our marketing reporting system—and how you can do the same.
The Challenge: Scaling Reporting Without Breaking the Bank
Initially, Funnel.io provided a seamless experience with its drag-and-drop interface, allowing us to integrate multiple marketing channels like Instagram, Facebook, Google Ads, and Apple Search Ads. It enabled us to build dashboards quickly and monitor our campaigns in real time. However, as our marketing efforts scaled, so did the costs. Each new dashboard added another layer of expenses, making it unsustainable.
I recall reviewing our monthly expenses late at night, realizing that while our marketing efforts fueled growth, reporting costs were eating into our margins. That’s when we began exploring open-source alternatives.
Discovering Airbyte: A New Direction
During a brainstorming session with our data team, one of our engineers introduced Airbyte—a modern, open-source data integration tool. Airbyte allowed us to extract data from various sources and load it into our data warehouse without the escalating fees of Funnel.io. The flexibility and active community support were key advantages.
Airbyte provided a growing library of pre-built connectors, enabling us to quickly integrate with key marketing channels:
Instead of paying more for each additional dashboard, we could configure and customize connectors as needed.
Building Our New Marketing Dashboard
Step 1: Data Extraction with Airbyte
We began by setting up Airbyte to extract raw data from our marketing channels:
- Deployment: We deployed Airbyte on our servers using Docker, making installation straightforward for our engineers.
- Connector Setup: Using Airbyte’s extensive library, we connected to our marketing channels and configured each connector to extract essential metrics like impressions, clicks, cost per click, and conversion rates.
- Automated Syncs: Airbyte’s scheduling features allowed us to set up automatic syncs, ensuring our data was always up to date without manual intervention.
This process provided efficiency and flexibility, giving us full control over our data without additional costs for every new connector.
Step 2: Centralizing Data in Google BigQuery
“To make the process easier, we created a diagram that illustrates the key steps for connecting Airbyte to BigQuery. Follow these steps as shown below:
Once extracted, our data was stored in Google BigQuery, a centralized, scalable data warehouse.
- Seamless Integration: Airbyte smoothly loaded our data into BigQuery, which is designed to handle vast amounts of data efficiently.
- Data Cleaning and Transformation: We performed minimal transformations to standardize data across channels, ensuring consistency in key performance indicators.
- Scalability: BigQuery’s architecture allowed us to manage growing data volumes effortlessly.
Step 3: Visualization with Looker Studio
With data centralized and cleaned in BigQuery, we used Looker Studio for visualization.
- Direct Connection: We connected Looker Studio directly to our BigQuery datasets, enabling real-time data access.
- Custom Dashboard Design: With Looker Studio’s drag-and-drop interface, we created dynamic dashboards displaying key marketing metrics such as ad spend, ROI, engagement rates, and conversion performance.
- Real-Time Insights: The dashboards refreshed in real time, ensuring our data was always up to date, allowing for quick, data-driven decisions.
The Benefits We Experienced
Transitioning from Funnel.io to our Airbyte + BigQuery + Looker Studio stack revolutionized our marketing reporting:
- Cost Efficiency: By leveraging open-source tools and scalable cloud solutions, we significantly reduced recurring costs, making expenses predictable and manageable.
- Flexibility: Our data engineers gained complete control over the ETL process, enabling full customization without vendor lock-in.
- Scalability: Google BigQuery efficiently handled our expanding data needs, while Looker Studio kept visualizations interactive and responsive.
- Real-Time Insights: Automated syncs and live data updates allowed for immediate decision-making, optimizing our marketing strategies on the fly.
Lessons Learned and Future Roadmap
Building our custom marketing dashboard taught us valuable lessons:
- Embrace Flexibility: Open-source tools like Airbyte provide the freedom to innovate without restrictive pricing models.
- Invest in Scalability: A scalable data warehouse like BigQuery ensures infrastructure isn’t a bottleneck.
- Prioritize Real-Time Data: Access to live data enhances agility in marketing decision-making.
- Empower Your Team: Giving data engineers control over ETL processes fosters innovative solutions tailored to business needs.
Looking ahead, we plan to refine our dashboards further and integrate additional data sources for deeper insights. Our journey is just beginning, with endless possibilities ahead.
Ready to Transform Your Marketing Reporting?
If you’re struggling with high reporting costs and inflexible dashboards, it may be time to rethink your data strategy. At Appgain, we built a solution that reduces costs while delivering real-time, actionable insights—helping us stay ahead in a competitive market.
Are you ready to build a custom marketing funnel, reporting dashboard, and growth engine tailored to your business? Contact us for a demo and let’s explore how we can turn your marketing data into a powerful asset.
