N. Okoro
← All projects

DATA ENGINEERING · STREAMING

Election Streaming Pipeline

How can a stream of simulated votes become continuously updated analytical views?

PythonSQLData EngineeringKafkaSparkStreamlitDocker

Simulated voting events · Portfolio case study

IMPLEMENTATION MAP · SIMULATED EVENTS

Python producers

Generate profiles and assign simulated votes

PostgreSQL

Store candidates, voters, and votes

Kafka → Spark

Consume vote events and aggregate by candidate and location

Kafka → Streamlit

Read aggregate topics for dashboard views

Architecture derived from supplied code · execution unverified

The question & data

An educational event-streaming implementation connects Python producers, Kafka topics, Spark aggregations, PostgreSQL storage, and a Streamlit dashboard.

Synthetic voter and candidate profiles are requested from Random User. The producer is configured for 1,000 voter-generation iterations and three candidates; these are configuration values, not verified processed totals.

The approach

  1. The Python producer creates database tables and publishes voter records. A second script randomly assigns candidates, stores votes, and publishes vote events.
  2. Spark reads votes_topic with an explicit schema, sets an event-time watermark, and writes candidate and location aggregates to separate Kafka topics with checkpoints.
  3. The Streamlit app reads aggregate topics and PostgreSQL counts, with refresh controls, charts, and a paginated location table. Docker Compose defines Kafka, ZooKeeper, and PostgreSQL services.

What the work reveals

The code separates event generation, persistence, stream processing, and presentation into distinct components.

Candidate totals and location counts are separate aggregate outputs, allowing different dashboard views from the same event stream.

From findings to next steps

Complete an end-to-end smoke test after repairing the entry script, moving local settings into configuration, and handling empty dashboard results. Validate replay behaviour and database-to-Kafka delivery consistency.

Read the results in context

The supplied main.py has a missing colon at line 116, preventing startup. Other Python files pass syntax parsing, but services were not run. Machine-specific Spark paths require configuration. This is a simulation, not a secure election system. No throughput, latency, or exactly-once guarantee has been established.

Explore the Spark aggregation
votes_per_candidate = enriched_votes_df.groupBy(
    "candidate_id", "candidate_name", "party_affiliation", "photo_url"
).agg(_sum("vote").alias("total_votes"))

Learning resource: Tutorial linked in the supplied README ↗

Based on supplied portfolio materials. Findings describe the dataset and report; recommendations are proposed next steps, not measured business outcomes.

Next case study Medallion Lakehouse & dbt Models ↗