Writing Dask partitions into single file

Question

New to dask,I have a 1GB CSV file when I read it in dask dataframe it creates around 50 partitions after my changes in the file when I write, it creates as many files as partitions.
Is there a way to write all partitions to single CSV file and is there a way access partitions?
Thank you.

Asked By: rey

||

Source

Answer 1

Short answer

No, Dask.dataframe.to_csv only writes CSV files to different files, one file per partition. However, there are ways around this.

Concatenate Afterwards

Perhaps just concatenate the files after dask.dataframe writes them? This is likely to be near-optimal in terms of performance.

df.to_csv('/path/to/myfiles.*.csv')
from glob import glob
filenames = glob('/path/to/myfiles.*.csv')
with open('outfile.csv', 'w') as out:
    for fn in filenames:
        with open(fn) as f:
            out.write(f.read())  # maybe add endline here as well?

Or use Dask.delayed

However, you can do this yourself using dask.delayed, by using dask.delayed alongside dataframes

This gives you a list of delayed values that you can use however you like:

list_of_delayed_values = df.to_delayed()

It’s then up to you to structure a computation to write these partitions sequentially to a single file. This isn’t hard to do, but can cause a bit of backup on the scheduler.

Edit 1: (On October 23, 2019)

In Dask 2.6.x, there is a parameter as single_file. By default, It is False. You can set it True to get single file output without using df.compute().

For Example:

df.to_csv('/path/to/myfiles.csv', single_file = True)

Reference: Documentation for to_csv

Answered By: MRocklin

Answer 2

you can convert your dask dataframe to a pandas dataframe with the compute function and then use the to_csv. something like this:

df_dask.compute().to_csv(‘csv_path_file.csv’)

Answered By: Fernando Siqueira

Writing Dask partitions into single file

Question:

Answers:

Short answer

Concatenate Afterwards

Or use Dask.delayed