Optimizing joins techniques in Apache Spark

Hi Readers,

In this post you will be learning the various optimizing joins techniques that can be used in Apache Spark.

Three types of joins in Spark are

Shuffle hash join (default):

  • It is a map-reduce type join.
  • Based on output key it shuffles the datasets.
  • During reduce phase, it joins the datasets for same output key.

Broadcast hash join:

  • Use this when one dataset is small enough to fit in memory.

Cartesian join:

  • Use this when every row of one table is joined with every row of the other table.

The below code snippet uses the .join() with orderDF and orderItemDF to get a joinedOrderDataDF.

scala> var joinedOrderDataDF = orderDF.join(orderItemDF,orderDF(“order_id”)===orderItemDF(“order_item_order_id”))
joinedOrderDataDF: org.apache.spark.sql.DataFrame = [order_id: int, order_date: bigint … 8 more fields]

check which execution strategy of the join has been used in joinedOrderDataDF using
scala>joinedOrderDataDF.explain —> to get which join type has been used.
scala>joinedOrderDataDF.queryExecution.executedPlan —> gives information on how the dataframe has been executed.


You can also bookmark this page for future reference.

You can share this page with your friends.

Follow me Jose Praveen for future notifications.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.