![]() This example uses Metabase deployed to Heroku. ![]() Once the data is in Redshift, we can write ad-hoc queries and visualize the data using trend analysis and data dashboards using a SQL-compliant analytics tool. The other Heroku app, reshift_batch 1, consumes events from Kafka and stores all the data in RedShift, which Amazon describes as "a fast, fully-managed, petabyte-scale data warehouse." viz shows the relative volume of product data being written into Kafka. ![]() A web-based data visualization app running on Heroku, viz 1, allows viewing the data flowing through Kafka in near real-time. In our example system, there are two apps that are downstream consumers of the data. Apache Kafka is an append-only immutable event log and the leading open source project for managing billions of events. The event stream is then available to other downstream consumers. Generate_data 1 is an app that produces a stream of events into an Apache Kafka cluster managed by Heroku. And more than one data producer can be added. The system uses a simple Node.js app deployed to Heroku called generate_data to simulate usage data for an e-commerce store, but this could be replaced with almost anything that produces data: a marketing website, a SaaS product, a point-of-sale device, a kiosk, an internet-connected thermostat or car. This is an example system that captures a large stream of product usage data, or events, to provide both real-time data visualization and SQL-based data analytics. We’ll also see how you can build consumers that can push those event streams from Heroku into Amazon Redshift for further analysis using Metabase, an open source product analytics tool. In this post, we’ll show you how to build a system using Apache Kafka on Heroku to manage and visualize event streams from any type of data producer. While this type of stream visualization is useful to a point, pushing events into a data warehouse lets you ask deeper questions using SQL. Managing event streams lets you view, in near real-time, how users are interacting with your SaaS app or the products on your e-commerce store this is interesting because it lets you spot anomalies and get immediate data-driven feedback on new features. Senior Director, Product Marketing December 06, 2018īuilding a SaaS product, a system to handle sensor data from an internet-connected thermostat or car, or an e-commerce store often requires handling a large stream of product usage data, or events.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |