Edge Analytics Pipeline

Problem

The product collected clickstream events through a single synchronous endpoint that wrote straight to Postgres. At a few hundred thousand events a day it worked; past a million it fell over — write contention, slow dashboards, and lost events during spikes.

What was built

• A thin ingestion service at the edge that validates events and pushes them onto a queue, acknowledging in well under 10ms.
• A batching consumer that writes rolled-up aggregates to Postgres on a fixed interval, keeping raw events in cheaper cold storage.
• A read path backed by Redis for the “last 24 hours” views, falling back to Postgres for historical queries.
• Backpressure and dead-letter handling so a downstream stall degrades gracefully instead of dropping data.

Outcome

Sustained ~4M events/day with dashboard queries consistently under 500ms, and no event loss during traffic spikes. Infrastructure cost per event dropped by roughly 60%.