Build Real-Time Dashboard Data Feeds

Real-Time Dashboard Feed Expert is a skill for designing low-latency data pipelines that power live dashboards and real-time analytics. Its architecture principles: build around events and state changes (event-driven architecture), implement backpressure handling to prevent overload, accept eventual consistency for better throughput, design operations to be safely retried (idempotency), and choose stateful versus stateless processing per requirement. Data-flow patterns covered include push delivery via WebSockets/SSE versus simple polling for pull scenarios, fan-out from a single source to many consumers, time-based/count-based/session-based aggregation windows, and event sourcing (storing events as an immutable, replayable log).

Use it when choosing and wiring a message-broker/delivery stack for streaming dashboard updates. It compares three brokers directly:

### Apache Kafka - High throughput, persistent
kafka:
  use_cases: ["high_volume", "persistent_storage", "complex_routing"]
  latency: "2-10ms"
  throughput: "millions/sec"

### Redis Streams - Low latency, simple
redis:
  use_cases: ["low_latency", "simple_setup", "caching"]
  latency: "<1ms"
  throughput: "hundreds_of_thousands/sec"

### Apache Pulsar - Multi-tenancy, geo-replication
pulsar:
  use_cases: ["multi_tenant", "geo_distributed", "unified_messaging"]

It is not the right fit for simple, infrequently-updated dashboards where periodic polling and a plain REST refresh would suffice - the streaming/WebSocket architecture here is built for high-frequency, low-latency updates.

Input is raw events from producers; output is live dashboard updates delivered to subscribed clients. On the delivery side it gives a Socket.io + Redis pub/sub server pattern that emits updates to per-dashboard rooms and sends initial state on subscribe, plus a client reconnection class with exponential backoff (up to 5 attempts) and requestAnimationFrame-batched update flushing to avoid excessive DOM writes. On the processing side it gives a Kafka Streams Java topology that windows raw events into 10-second aggregates (TimeWindows.of(Duration.ofSeconds(10))) and publishes them to a dashboard-updates topic, and a Python pipeline that consumes from Kafka, caches the latest value per metric in Redis, and broadcasts to connected WebSocket clients while pruning disconnected ones. For payload efficiency it recommends MessagePack serialization with gzip compression applied only above a 1024-byte threshold.

For observability it wires in Prometheus client metrics - a Counter for messages processed per dashboard, a Histogram for processing latency, and a Gauge for active WebSocket connections per dashboard. For resilience it implements a circuit breaker (CLOSED/OPEN/HALF_OPEN states, configurable failure threshold and reset timeout) around downstream calls to stop cascading failures. Broader scalability guidance covers horizontal scaling behind load balancers, database read replicas to separate read/write load, Redis/Memcached caching layers, connection pooling, and rate limiting.

Backend and platform engineers building the streaming layer behind live dashboards or monitoring systems, who need both the transport/processing architecture and its operational hardening: JWT or session-based auth on WebSocket connections, per-client and per-dashboard rate limits, input validation on all incoming data, correct CORS configuration, and SSL/TLS for data in transit.