PySpark Optimization cheat sheet with Shrividya Hegde

A two-page reference for the PySpark decisions that actually change your runtime: partitioning, joins, shuffles, and skew. Current through Spark 4.0.

Most PySpark advice online is either a beginner tutorial or a wall of config flags with no guidance on when to use them. This is neither. It's the reference I reach for when a job is slow and I need to know what to check first.

Two pages, eleven sections, built for your second monitor:

• Partitioning: repartition vs coalesce, and how to size partitions
• Joins: broadcast vs sort-merge, with a diagram showing why broadcast wins
• Caching: storage levels, and the PySpark-specific gotcha that Scala docs get wrong for Python users
• Shuffle avoidance, skew handling with AQE and salting
• Spark UI diagnostics: which columns tell you it's skew, spill, or GC pressure
• explain() red flags, pandas UDFs, and a full config table with copy-paste able keys

Includes a Spark 4.0 note on ANSI mode now defaulting to on, which turns silent nulls into runtime errors. Worth knowing before you upgrade.