spark-optimization

Workload-aware ML framework for PySpark performance prediction and configuration recommendation, validated through real benchmark execution and out-of-distribution evaluation.

Apache Spark native computation engine

For the GitHub description field: "Hands-on PySpark performance optimization repo — broadcast joins, partitioning, caching, bucketing, AQE, and skew handling, each with runnable before/after benchmarks and correctness tests.

Demonstrates PySpark broadcast joins vs regular joins in Databricks, with execution plan comparison for performance optimization.

This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.

⚡ Intelligent Data Skew Detection & Mitigation in Apache Spark — ML Severity Classifier + Automated Salting/Repartitioning