Spark Window functions using parititionby and order by on department dataset

on

 How to get rank dept wise in spark using parititionby and order by.

Input:

id,name,dept,salary

1,Shiva,Electrical,90000

2,Ram,Electrical,85000

3,Mohan,Electrical,95000

4,Raju,Electrical,96000

1,Shiva1,Computers,91000

2,Ram1,Computers,80000

3,Mohan1,Computers,97000

4,Raju1,Computers,90000


Code:

Let suppose we have a dataframe with name df then perform rank and get ranks for each person in each department.

ranks = Window.partitionBy("dept").orderBy(df["salary"].desc())
result_df = df.withColumn("ranks",rank().over(ranks))
result_df.show()



Comments

Post a Comment