pandas to_csv doesn't output the file
Question:
I’m using jupyter notebook pandas to_csv doesn’t output the dataframe to a file.
I tried to use to_csv to output a dataframe to a csv file by setting the working directory or specify the directory, and it didn’t create any file. The code ran and didn’t produce any error message, but when I opened the folder, there was no such file.
I tried a different IO, and it did show the result had been output.
from io import StringIO
output = StringIO()
a.to_csv(output)
print(output.getvalue())
I got the following output:
,a
0,1
1,2
2,3
but again to_csv('filepath/filename.csv') doesn’t output any file.
PS: I can read any file in any directory using read_csv().
Update
If I save the file df.to_csv('testfile.csv') then do pd.read_csv('testfile.csv')
I can read the file but cannot see it in the directory.
Also, doing [x for x in os.listdir() if x == 'testfile.csv'] will list the file.
Answers:
Maybe you do not have access to your output folder.
First try the current dir, like to_csv('tmp.csv').
Then check the directory’s ownership by using ls -l.
you probably forgot to add the name of the file after your path, so it will named your file as the last character of your path, which you can see on the home page of jupyter.
should be:
df.to_csv(‘path/filename.csv’, ….)
rather than
df.to_csv(‘path.csv’……)
I think the issue is that you’re running a Jupyter Notebook so the “current directory” for the notebook is probably somewhere in “C:Usersuser_nameAppData…”.
Try running os.getcwd() on its own in your notebook. It probably won’t be the same folder as where the *.ipynb file is saved. So as @Chris suggested in comments, this:
df.to_csv(os.getcwd()+'\file.csv')
… will send your csv into the AppData folder.
You could either change the working directory for the Jupyter notebook, or you could use a fully specified filename like:
df.to_csv('C:\Users\<user_name>\Desktop\file.csv')
(Note: this also tripped me up in VS-Code while using the interactive iPython execution that happens when you press shift+enter. Interestingly, in VS-Code, if you use ctrl+shift+p and select “Python: Run selection…” it executes in your default terminal, which doesn’t have this problem.)
I had the same problem using spyder. In my case it was caused by the internet security tool (COMODO) I used, which somehow executed spyder in a sandbox or so and not allowed it to write to the “normal” directories. Instead it saved the result to a folder named C:VTRootHarddiskVolume2users. You can try to save a file with a quite unique name a.to_csv(‘very_unique_filename.csv’) and then search in the windows explorer for that filename to find the folder it is stored in. If the reason is some tool like this, changing it’s settings may help.
This will give you a normal document even though you have used to_csv:
df.to_csv(**'df',** index = False)
Make sure to use ‘df.csv’ this will ensure the output CSV file.
df.to_csv(**'df.csv'**, index = False)
in my case the files were saved to my apps root folder (as expected).
but i needed to restart Jupyter even though i had auto reload enabled:
%load_ext autoreload
%autoreload 2
meaning, reload works for most changes, but it did not work when i added df.to_csv until restarting jupyter notebook
You may be using PyCharm for some reason when you create a new CSV it doesn’t show the created file in the tree, nevertheless, it was created, you should check the file in the terminal. Or simply close and reopen PyCharm.
I’m using jupyter notebook pandas to_csv doesn’t output the dataframe to a file.
I tried to use to_csv to output a dataframe to a csv file by setting the working directory or specify the directory, and it didn’t create any file. The code ran and didn’t produce any error message, but when I opened the folder, there was no such file.
I tried a different IO, and it did show the result had been output.
from io import StringIO
output = StringIO()
a.to_csv(output)
print(output.getvalue())
I got the following output:
,a
0,1
1,2
2,3
but again to_csv('filepath/filename.csv') doesn’t output any file.
PS: I can read any file in any directory using read_csv().
Update
If I save the file df.to_csv('testfile.csv') then do pd.read_csv('testfile.csv')
I can read the file but cannot see it in the directory.
Also, doing [x for x in os.listdir() if x == 'testfile.csv'] will list the file.
Maybe you do not have access to your output folder.
First try the current dir, like to_csv('tmp.csv').
Then check the directory’s ownership by using ls -l.
you probably forgot to add the name of the file after your path, so it will named your file as the last character of your path, which you can see on the home page of jupyter.
should be:
df.to_csv(‘path/filename.csv’, ….)
rather than
df.to_csv(‘path.csv’……)
I think the issue is that you’re running a Jupyter Notebook so the “current directory” for the notebook is probably somewhere in “C:Usersuser_nameAppData…”.
Try running os.getcwd() on its own in your notebook. It probably won’t be the same folder as where the *.ipynb file is saved. So as @Chris suggested in comments, this:
df.to_csv(os.getcwd()+'\file.csv')
… will send your csv into the AppData folder.
You could either change the working directory for the Jupyter notebook, or you could use a fully specified filename like:
df.to_csv('C:\Users\<user_name>\Desktop\file.csv')
(Note: this also tripped me up in VS-Code while using the interactive iPython execution that happens when you press shift+enter. Interestingly, in VS-Code, if you use ctrl+shift+p and select “Python: Run selection…” it executes in your default terminal, which doesn’t have this problem.)
I had the same problem using spyder. In my case it was caused by the internet security tool (COMODO) I used, which somehow executed spyder in a sandbox or so and not allowed it to write to the “normal” directories. Instead it saved the result to a folder named C:VTRootHarddiskVolume2users. You can try to save a file with a quite unique name a.to_csv(‘very_unique_filename.csv’) and then search in the windows explorer for that filename to find the folder it is stored in. If the reason is some tool like this, changing it’s settings may help.
This will give you a normal document even though you have used to_csv:
df.to_csv(**'df',** index = False)
Make sure to use ‘df.csv’ this will ensure the output CSV file.
df.to_csv(**'df.csv'**, index = False)
in my case the files were saved to my apps root folder (as expected).
but i needed to restart Jupyter even though i had auto reload enabled:
%load_ext autoreload
%autoreload 2
meaning, reload works for most changes, but it did not work when i added df.to_csv until restarting jupyter notebook
You may be using PyCharm for some reason when you create a new CSV it doesn’t show the created file in the tree, nevertheless, it was created, you should check the file in the terminal. Or simply close and reopen PyCharm.