PySpark Tutorial for Beginners: Install, DataFrames, Joins, Window Functions and Parquet
A step-by-step PySpark tutorial from pip install to a Parquet file. Each step is one short example you can run on your laptop or in PrepNPlaced's Coding Practice: a SparkSession, a CSV, select and filter, new columns, groupBy, joins without duplicate columns, window functions and Parquet.
By Durgesh Yadav — Senior Data Engineer @ 7-Eleven · Updated 15 Oct 2026. Preparation guidance, not a hiring guarantee.
How do I start learning PySpark as a beginner?
Install Java 17 or later and run pip install pyspark, as the Spark 4.2.0 install guide requires, or skip setup and use PrepNPlaced's Coding Practice, where a Spark session is ready. Then learn in this order: read a CSV, select and filter, add columns, group and aggregate, join, use window functions, and write Parquet. Type each step yourself and check it with show() before moving on.
Install Java first, then PySpark with pip. The PySpark install guide for Spark 4.2.0 supports Python 3.10 and above and says PySpark needs Java 17 or later with JAVA_HOME set; the main Spark docs list Java 17, 21 and 25. The pip package carries Spark itself, so you do not download a separate Spark release for local work. Use a virtual environment so the version stays with the project.
1Check Python with python3 --version and confirm it is 3.10 or newer.
2Install a Java 17 or Java 21 JDK, both supported by Spark 4.2.0, and check it with java -version.
3Set JAVA_HOME to the JDK folder, in your shell profile on macOS or Linux or in the environment variables settings on Windows.
4Create a virtual environment with python3 -m venv .venv and activate it.
5Run pip install pyspark inside that environment.
6Type pyspark in the terminal: a shell that prints the Spark version and says a SparkSession is available as spark means the install worked.
Can you run PySpark in VS Code?
Yes. VS Code runs PySpark like any Python script, as long as the interpreter it uses is the one where you ran pip install pyspark and Java is on the PATH or in JAVA_HOME. Open the project folder, choose the .venv interpreter with the Python: Select Interpreter command, and run the file. Jupyter notebooks inside VS Code work the same way once the kernel is that environment. While the session is alive, the Spark UI is at http://localhost:4040.
To run against a Databricks cluster instead of your laptop, Databricks publishes an extension for Visual Studio Code and Cursor that runs local Python files on Databricks clusters or serverless compute.
JAVA_HOME is not set, or Java gateway process exited: Java is missing or JAVA_HOME points to the wrong folder
No module named 'pyspark': VS Code is using a different interpreter from the one you installed into
Port 4040 already in use: another session is open, so Spark moves the UI to 4041
How do you create a SparkSession and read a CSV?
Every PySpark program starts from a SparkSession, the entry point to DataFrames and SQL. In a script, write from pyspark.sql import SparkSession and then spark = SparkSession.builder.appName('tutorial').getOrCreate(); the pyspark shell, Databricks notebooks and Coding Practice create it for you. header=True takes column names from the first line, and inferSchema=True makes Spark scan the data to guess types. For a production job, pass an explicit schema, and add mode='FAILFAST' if a malformed row should stop the job.
Read a CSV and look at it
orders = spark.read.csv('orders.csv', header=True, inferSchema=True); orders.printSchema(); orders.show(5). printSchema lists each column and its type; show(5) prints five rows. In the steps below, orders has order_id, customer_id, city, amount and order_date, and customers has customer_id, name and city.
How do you select, filter and add columns?
select picks columns, filter keeps the rows that match a condition, and withColumn adds or replaces one column. Build conditions with F.col and combine them with & and |, each side in brackets, because Python's and and or do not work on columns. Since Spark 4.0, ANSI mode is on by default, so casting a bad string such as 'abc' to a number raises an error instead of returning null; when you expect bad values, use try_cast, which returns null for them.
Filter, then add a column
pune = orders.select('order_id', 'city', 'amount').filter((F.col('amount') > 500) & (F.col('city') == 'Pune')); pune = pune.withColumn('band', F.when(F.col('amount') >= 2000, 'high').otherwise('normal'))
Cast a messy text column safely
raw = raw.withColumn('amount', F.col('amount').try_cast('double')). Rows where amount is not a number get null instead of failing the job; count them with raw.filter(F.col('amount').isNull()).count().
How do you group, aggregate and join without duplicate columns?
groupBy followed by agg is PySpark's GROUP BY: name the key columns, then one F function per output column, with alias to name it. For joins, pass the key as a name, on='customer_id', or a list for several keys, on=['city', 'order_date']. Per the DataFrame.join docs, those names must exist on both sides and the join is an equi-join; the result keeps one copy of each key. Join on an expression such as orders.customer_id == customers.customer_id and you get two customer_id columns, and a later select('customer_id') fails as ambiguous.
The name form merges only the key. If both tables also have a city column, you get two, so select the columns you need from the right side or rename one first. The how argument sets the join type: inner by default, plus left, right, full, left_semi and left_anti among others.
joined = orders.join(customers.select('customer_id', 'name'), on='customer_id', how='left'). The result has one customer_id, and selecting only name from customers keeps its city from clashing with orders.city. If an expression join already gave you two keys, joined.drop(customers.customer_id) removes the right-hand copy.
How do you use window functions and write Parquet?
A window function computes a value for each row from a group of related rows, without collapsing them the way groupBy does. Define the window with Window.partitionBy for the groups and orderBy for the order inside each, then apply F.row_number, F.rank, F.lag or a running F.sum over it. The classic interview prompt is top N per group.
Then write the result as Parquet, a columnar format that stores the schema with the data. mode('overwrite') replaces earlier output, and partitionBy writes one folder per value, so later reads can skip folders they do not need. The write is an action: this is the line where every earlier step runs.
Top two orders per city
w = Window.partitionBy('city').orderBy(F.desc('amount')); top2 = orders.withColumn('rn', F.row_number().over(w)).filter(F.col('rn') <= 2).drop('rn')
Write Parquet, partitioned by city, and read it back
top2.write.mode('overwrite').partitionBy('city').parquet('output/top_orders'); spark.read.parquet('output/top_orders').printSchema(). The schema comes back from the files, and city returns as a column built from the folder names.
One that makes you type and run every line. The official PySpark Quickstart for DataFrames on spark.apache.org is short and correct, and this page follows the same order with joins, windows and Parquet added. After that, graded questions teach more than further reading.
Do I need Hadoop installed to run PySpark locally?
No. The pip package carries the Hadoop client libraries Spark uses, and local mode runs the driver and executors in one process on your machine. You need Python 3.10 or above and Java 17 or later with JAVA_HOME set, per the Spark 4.2.0 install guide.
What is the difference between select and withColumn?
select returns only the columns you list, and you can compute new ones inside it. withColumn keeps every column and adds or replaces one. For many new columns, use one select: the withColumn docs warn that calling it in a loop builds very large plans.
Why does my join return more rows than the left table?
The right table has more than one row for some keys, and a join returns one row per match. Check before joining with customers.groupBy('customer_id').count().filter(F.col('count') > 1). Fix it by removing the duplicates on purpose, for example keeping the latest row per key with a window, and then join.
Can I practise PySpark online without installing it?
Yes. PrepNPlaced's Coding Practice has a PySpark track where spark, F and Window are ready. Running code is free; Run shows the first 50 rows of your result, and after signing in, Submit grades it against the expected output.
Next Step
Turn The Guide Into Practice
Use PrepNPlaced tools to turn this learning path into resume proof, targeted practice, and interview-ready explanations.
