We are making a complete collection of 2019 Spark interview questions and Apache Spark tutorial.This video is an addition to the collection. In this video we walk through Apache Spark Cost Based Optimizer aka CBO in Spark example.

As part of the example we create 3 tables and apply inner join between them and run two kind of queries.

in one case:-
We run inner joins between three tables when CBO is turned off.

We run same query when CBO is turned on and once we have collected required statistics and compare query plans in these two cases.

In other kind of query:- .

We run inner joins between three tables with filter operation on one of the table when CBO is turned off.

We run same query when CBO is turned on and once we have collected required statistics and compare query plans in these two cases.

We discuss how spark opts for shuffle hash join,sort merge join and broad cast join in different cases and fatcors to be considered while using CBO.

CBO can easily be counted as one of the best practices to be followed and we can boost performance of spark application without much effort from user's end.

CBO is enhancement to Spark Catalyst and is introduced in Spark 2.2.In Spark 2.1 Spark Catalyst is rule based and in most of the cases achieves sub optimal plan.Other than physical planning phases all phases of Catalyst Optimizer is rule based and it does not consider table/column statistics and cost per operator while applying optimization.

CBO :-    • Spark Cost Based Optimizer (CBO) | Sp...  
Structtype and structfield :-    • StructType & StructField | Create Dat...  

About channel
---------------------
We are trying to make video series on technical interview questions on various skills for experienced and freshers. This is based on the industry experience. We are trying to compile all the

possible questions with answers and give you the core concepts of the topic.

We will present a series on tutorial on various skill sets as well. Our first tutorial will be on Apache Spark and we will present it soon.

Our aim is to make a difference in your learning.

Please subscribe to our channel and stay connected.

Thank you so much.

e-mail:- mannabunty3@gmail.com


Our video links
________________

Cost Based Optimizer Example:-    • Video  

Cost Based Optimizer :-    • Video  

Spark StructType & StructField:    • StructType & StructField | Create Dat...  

Spark application's life cycle - 2 :    • Spark Execution Flow - 2 |Spark Sessi...  
------------------------------------

Spark application's life cycle - 1 :    • Video  
------------------------------------

Parquet with Spark :    • Apache Parquet and Apache Spark | Spa...  
--------------------------------

Type safety in Dataframe vs Dataset - Part 2 :    • Type Safety in Dataset vs Dataframe (...  
----------------------------------------------------------------------

Type safety in Dataframe vs Dataset - Part 1 :    • Type Safety in Dataset vs Dataframe (...  
----------------------------------------------------------------------

Broadcast variable example :    • Broadcast variable example | Spark In...  
---------------------------------------------

Broadcast variable vs Normal variable :    • Broadcast variable in Spark |Shared v...  
-------------------------------------------------------------

RDD vs Dataframe vs Dataset :    • RDD vs Dataframe vs Dataset | Spark I...  
-----------------------------------------------

toDF() vs createDataFrame() :    • Spark Interview Questions and Answers...  
----------------------------------------------

User defined function Part -1 :    • UDF ( Part 1) |  User defined functio...  
---------------------------------------------

User defined function Part - 2 :    • UDF (Part 2) | user defined function ...  
----------------------------------------------

Dataframe :    • What is Dataframe | Dataframe API | S...  
-----------------

What is Apache Spark :    • What is Spark | Spark interview quest...  
------------------------------------