Generate profiles and assign simulated votes
DATA ENGINEERING · STREAMING
Election Streaming Pipeline
How can a stream of simulated votes become continuously updated analytical views?
Simulated voting events · Portfolio case study
IMPLEMENTATION MAP · SIMULATED EVENTS
Store candidates, voters, and votes
Consume vote events and aggregate by candidate and location
Read aggregate topics for dashboard views
The question & data
An educational event-streaming implementation connects Python producers, Kafka topics, Spark aggregations, PostgreSQL storage, and a Streamlit dashboard.
Synthetic voter and candidate profiles are requested from Random User. The producer is configured for 1,000 voter-generation iterations and three candidates; these are configuration values, not verified processed totals.
The approach
- The Python producer creates database tables and publishes voter records. A second script randomly assigns candidates, stores votes, and publishes vote events.
- Spark reads votes_topic with an explicit schema, sets an event-time watermark, and writes candidate and location aggregates to separate Kafka topics with checkpoints.
- The Streamlit app reads aggregate topics and PostgreSQL counts, with refresh controls, charts, and a paginated location table. Docker Compose defines Kafka, ZooKeeper, and PostgreSQL services.
What the work reveals
The code separates event generation, persistence, stream processing, and presentation into distinct components.
Candidate totals and location counts are separate aggregate outputs, allowing different dashboard views from the same event stream.
From findings to next steps
Complete an end-to-end smoke test after repairing the entry script, moving local settings into configuration, and handling empty dashboard results. Validate replay behaviour and database-to-Kafka delivery consistency.
Read the results in context
The supplied main.py has a missing colon at line 116, preventing startup. Other Python files pass syntax parsing, but services were not run. Machine-specific Spark paths require configuration. This is a simulation, not a secure election system. No throughput, latency, or exactly-once guarantee has been established.
Explore the Spark aggregation
votes_per_candidate = enriched_votes_df.groupBy(
"candidate_id", "candidate_name", "party_affiliation", "photo_url"
).agg(_sum("vote").alias("total_votes"))Learning resource: Tutorial linked in the supplied README ↗
Based on supplied portfolio materials. Findings describe the dataset and report; recommendations are proposed next steps, not measured business outcomes.