Read/Write single file in DataBricks

Question

I have a file which contains a list of names stored in a simple text file. Each row contains one name. Now I need to pro grammatically append a new name to this file based on a users input.
For the input itself I use DataBricks widgets – this is working just fine and I have the new name stored in a string object.
Now I need to append this name to my file.

the file is mounted in the DataBricks File System (DBFS) under /mnt/blob/myNames.txt

when trying to read the file like this:

f = open("/mnt/blob/myNames.txt", "r") 
print f

it returns an error “No such file or directory”

So I tried to wrap my new name into a dataframe and append it to the existing file but this also did not work as dataframe.write.save is designed to write into folders

what would be the most simple python could that I could use to append this new name to my file?

Asked By: Gerhard Brueckl

||

Source

Answer 1

You can write and read files from DBFS with dbutils. Use the dbutils.fs.help() command in databricks to access the help menu for DBFS.

You would therefore append your name to your file with the following command:

dbutils.fs.put("/mnt/blob/myNames.txt", new_name)

You are getting the “No such file or directory” error because the DBFS path is not being found. Use dbfs:/ to access a DBFS path. This is how you should have read the file:

f = open("/dbfs/mnt/blob/myNames.txt", "r")

Answered By: Elsis

Answer 2

You can open the file in append mode using ‘a’

with  open("/dbfs/mnt/sample.txt", "a") as f:
  f.write("append values")

Now you can view the contents using read mode ‘r’

with  open("/dbfs/mnt/sample.txt", "r") as f_read:
  for line in f_read:
    print(line)

Solution: Here

Answered By: USB

Read/Write single file in DataBricks

Question:

Answers: