Skypulse Streaming Pipeline

SkyPulse is an end-to-end real-time data pipeline that integrates heterogeneous global streams—flight positions, weather conditions, and seismic events—into a unified operational view of airspace. It continuously ingests data from OpenSky, Open-Meteo, and USGS, processes it through Redpanda and Apache Flink using 5-minute tumbling windows, and stores it in a Supabase landing zone. The core strength of the system lies in its use of Bruin, which structures and transforms raw data into a layered analytical model (staging → intermediate → marts), enabling consistent, cross-stream enrichment. This allows the generation of composite geospatial risk scores per grid cell, combining aircraft density, seismic activity, and weather conditions, all visualized in near real-time through an interactive Streamlit dashboard