Joins what shoppers browsed with what they bought, in one governed lakehouse pipeline.
Most retail data stacks keep behavior and transactions apart. What shoppers did (page views, cart adds, sessions) lives in an event stream, and what they bought lives in the checkout database. Analysts stitch the two together late and inconsistently, yet that gap is exactly where the important questions live: conversion funnels, cart abandonment, and revenue by browsing behavior. StreamCart treats both as first-class inputs to one governed pipeline. Clickstream events arrive in near real time through Kafka and Spark Structured Streaming, orders arrive through scheduled batch extracts from Postgres, and both are conformed into a single Delta Lake medallion architecture with a shared star schema. One dashboard query can then answer "what's our conversion rate by category this week" with no ad hoc joining.
Faker seeds Postgres with 3,000 customers, 2,000 products and 10,000 orders across about 15 months. A session-aware producer streams clickstream events to Kafka (KRaft), where each session follows a real funnel: browse, add to cart, then abandon or purchase.
Spark Structured Streaming lands Kafka events with checkpoint-based exactly-once delivery, while JDBC batch jobs extract the OLTP tables. Both land as raw Delta tables.
Deduplication, schema enforcement and null handling, plus SCD Type 2 history on customer and product dimensions. Data-quality gates run inside the transforms and fail the task rather than just warning.
fact_orders and fact_events join dim_customer, dim_product and dim_date, with pre-aggregated daily active users, conversion funnel and revenue-by-category tables. dbt re-models the gold layer in tested, documented SQL and adds customer LTV and cohort retention marts.
An Airflow DAG runs bronze → silver → gold with retries and failure alerting. Gold tables are exported to Neon Postgres, and a Streamlit dashboard on Community Cloud reads them through DATABASE_URL.
Batch ingestion row counts match Postgres exactly (3,000 / 2,000 / 10,000 / 20,666 rows)
Re-running the stream with no new messages added zero rows: exactly-once delivery confirmed
Continuous mode picked up all 804 newly produced events across micro-batches with no loss or duplication
Injected bad data fails the silver gate, and Airflow retries then alerts, as designed
CI runs pytest plus a full dbt build on every push, using synthetic gold fixtures