Python Data Engineering Platform

Tech Stack

Python 3.12

Apache Spark

Apache Kafka

PostgreSQL

Docker

dbt

Features

✓

Real-time streaming with Kafka

✓

Batch processing with Spark

✓

CSV/API data ingestion

✓

Data transformation with dbt

✓

PostgreSQL data lake

✓

Docker containerization

Code Sample

spark_jobs.py

from pyspark.sql import SparkSession
from pyspark.sql.functions import from_json, col

spark = SparkSession.builder \
    .appName("DataEngineering") \
    .config("spark.jars.packages",
            "org.apache.spark:spark-sql-kafka") \
    .getOrCreate()

df = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "raw_events") \
    .load()

transformed = df.select(
    from_json(col("value").cast("string"), schema).alias("data")
).select("data.*")

transformed.writeStream \
    .format("jdbc") \
    .option("url", "jdbc:postgresql://localhost/analytics") \
    .start()