Writing to CSV, parquet, avro files from spark dataframe

Write a dataframe to a particular path

We have to two types of modes to write a file

• overwrite
• append

Writing the Parquet file using overwrite, append

df.write.format("parquet").mode("overwrite").save("path_to_save")

df.write.format("parquet").mode("append").save("path_to_save")

Writing to avro file using overwrite, append

df.write.format("avro").mode("overwrite").save("path_to_save")

df.write.format("avro").mode("append").save("path_to_save")

Writing to csv file using overwrite, append

Using comma as separator

df.write.option("header", "true").option("sep", ",").mode("overwrite").csv("path_to_save")

df.write.option("header", "true").option("sep", ",").mode("append").csv("path_to_save")

Using pipe as separator

df.write.option("header", "true").option("sep", "|").mode("overwrite").csv("path_to_save")

df.write.option("header", "true").option("sep", "|").mode("append").csv("path_to_save")