World Tech Cure

Search This Blog

How to read parquet files in spark

on September 05, 2021

Get link
Facebook
X
Pinterest
Email
Other Apps

Reading parquet files using spark

df = spark.read.parquet("path_to_parquet")

df.show()

spark

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

How to create setup.py file in python project

Create setup.py file in python project. We need to create a setup.py file in root directory of your project so that it will access the whole project. Template of setup.py file. When ever we want to create a pyspark project we have to create a executable file which can be uploaded into spark environment to run the code. To create that executable file we need a setup.py file which will be similar to pom file in java programs It will hold the what are the libraries we need in the project. In the script option we need to mention the main script will be called as an starting point of our project. Find packages will ping all the packages present in the code. from setuptools import setup, find_packages setup( name="project name", description="Description of project", version='1.0', author="author name", author_email="author email id", url="", packages=find_packages(), scripts=["hello_world.py"...

How to create Spark session in scala spark and pyspark

Create Spark session in scala spark and pyspark Scala-spark val spark = SparkSession.builder .master("local") .appName("My Application") .getOrCreate() PySpark spark = SparkSession.builder \ .master("local") \ .appName("My Application") \ .getOrCreate() We can use spark variable now to work with dataframes and perform all spark actions.

How to execute multiple lines of spark scala code at a time

Execute multiple lines of spark scala code at a time scala> :paste // Entering paste mode (ctrl-D to finish) val a = 10 // Exiting paste mode, now interpreting. a: Int = 10 scala> In the spark-shell we need to write :paste press enter Paste the code to be executed. Then click on ctrl-D then code will be executed and you can see the output

Powered by Blogger

How to read parquet files in spark

Comments

Post a Comment