Skip to main content

World Tech Cure

Search This Blog

Spark vs MapReduce

on September 14, 2021

Get link
Facebook
X
Pinterest
Email
Other Apps

Spark vs MapReduce

Differences between Spark and MapReduce

Spark preforms well to work with heavy work loads of data and to finish pipelines in less time.

spark

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

Post a Comment

Popular Posts

How to create Spark session in scala spark and pyspark

Create Spark session in scala spark and pyspark Scala-spark val spark = SparkSession.builder .master("local") .appName("My Application") .getOrCreate() PySpark spark = SparkSession.builder \ .master("local") \ .appName("My Application") \ .getOrCreate() We can use spark variable now to work with dataframes and perform all spark actions.

Optimizations in spark pipeline

Optimizations in spark pipeline We can handle optimizations in spark by following below scenarios: File formats Proper cache handling Persist Using DataFrame Using coalesce and Repartition Broadcast variables or tables Choose right partition column Memory allocation File formats: We need to make best use of existing file formats. Avro can be used to perform quick writes. Parquet used even we need quick reads, due to columnar storage we can read only required column from whole parquet file. Use Snappy compression. Proper cache handling: Cache the dataframe which is getting used multiple in same spark execution flow because it will prevent creating the same dataframe multiple times when ever it is accessed. Cache will store the dataframe in memory and provides us whenever required without recreating. Persist: Depending upon the cluster configuration we can use best storage level to persist dataframe in memory or disk. Using DataFrame: Dataframes has best optimizations when compared ...

Sparksession vs hivecontext vs sqlcontext vs sparkcontext

Sparksession vs hivecontext vs sqlcontext vs sparkcontext Apache spark has been became more popular data processing framework because of its flexibility and speed. During the process of evolution of spark with different versions there are significant changes. We need a entry point for any of the framework which helps program to start, with respect to entry point execution spark has different features. Till spark version <2.0 we had Sparkcontext, Hivecontext and SQLcontext We will the differences between them: Spark Context : It is the entry point of the spark applications where the driver program starts, driver program is the process which divides the application logic into tasks and assigns it to each executer which is available on worker node. Scala spark: val sparkConf = new SparkConf().setAppName("my app").setMaster("local[1]") val sparkContext = new SparkContext(sparkConf ) Pyspark: sparkConf = SparkConf().setAppName("my app").setMaster("loca...

Powered by Blogger