Related

Initialize Spark

pyspark

Create SparkSession to start PySpark

Create DataFrame

pyspark

Create DataFrame from list or RDD

Read CSV

pyspark

Load CSV file into DataFrame

pyspark/sql

Run SQL

Execute SQL queries

pyspark
sql
query

Command

spark.sql()

Explanation

Register DataFrame as temp view and run SQL queries.

Examples

Register DataFrame as SQL table

df.createOrReplaceTempView("people")

Run SQL query

spark.sql("SELECT * FROM people WHERE age > 25").show()

More

Show DataFrame

pyspark

Display rows of DataFrame

Print Schema

pyspark

Print schema of DataFrame

Select Columns

pyspark

Select specific columns from DataFrame