Why is pandas read_csv able to find file but os.listdir isn't?

Question

I have a file at ~/Google Drive/iNat/full.csv.

iNatPATH = r"~/Google Drive/iNat/"
FILENAME = iNatPATH + "full.csv"

I can’t figure out why Pandas is able to read the csv file:

import pandas as pd
raw_data = pd.read_csv(FILENAME)

# no error returned.

But os can’t find it:

import os
os.path.exists(FILENAME)

# outputs False

I am running this code in a macOS system on a local runtime in Google Colab.

My working directory is (not supposed to be relevant since I am using the absolute path):

>>> os.getcwd()
...
'/Users/<myusername>/Documents'

Asked By: willcrack

||

Source

Answer 1

Your problem is that os.path.exists() function doesn’t expand the ~ character. To solve this, the same os module provides the os.path.expanduser(path) function that will expand that user variable in a path for you.

Applied to your case:

iNatPATH = r"~/Google Drive/iNat/"
FILENAME = iNatPATH + "full.csv"

# Expand the ~ character
FILENAME = os.path.expanduser(FILENAME)

# Finally you can check again with exists() function
os.path.exists(FILENAME)

Reference: https://docs.python.org/3/library/os.path.html#os.path.expanduser

Answered By: Btc Sources

Answer 2

I ran into a similar problem, where os.list.dir() returned a path and then subsequently pd.read_csv() threw a FileNotFound error and alsoos.path.exists() returned False.

If anyone encounters the same error, check that your filename does not exceed 259 characters (Windows) or 255 characters (MacOS).

(While not a direct answer to the question, it’s very likely that individuals experiencing this problem may land on this stackoverflow page, as I did.)

Answered By: baileyw

Why is pandas read_csv able to find file but os.listdir isn't?

Question:

Answers: