Python conditional loop evaluating backwards

Question

So I’ve got this loop I’m working on, and it is supposed to check n number of csv files, look for "TD" in cell A1, and if it doesn’t find it move the file to the badPath. But the it evaluates exactly backwards, meaning that files with "TD" get dumped into the badPath. This is my ‘working’ code. With the shutil.move(os.path.join(path,f),badPath) call outside of the with loop the code executes, only it identifies files with this identifier as not having it.

data = []

for f in TD_files:    
    with open(f, newline='', encoding='utf-8') as g: 
        r = csv.reader((line.replace('', '') for line in g))
        data = [line for line in r] 
        if data[0] != 'TD': 
            print('Not A Trend File') 
    shutil.move(os.path.join(path,f),badPath)

Code I have tried which hasn’t worked:

data = []

for f in TD_files:    
    with open(f, newline='', encoding='utf-8') as g: 
        r = csv.reader((line.replace('', '') for line in g))
        data = [line for line in r] 
        if data[0] != 'TD': 
            print('Not A Trend File') 
            shutil.move(os.path.join(path,f),badPath)

This produces a PermissionError because the processes are all tied up in one another. I have included an image of the error and called it error1.

data = []

for f in TD_files:    
    with open(f, newline='', encoding='utf-8') as g: 
        r = csv.reader((line.replace('', '') for line in g))
        data = [line for line in r if data[0] != 'TD']  
        print('Not A Trend File')
        shutil.move(os.path.join(path,f),badPath)

This code was my first inclination and it produces an IndexError: list out of range which leads me to assume that my evaluations are incorrect.

Question

Am I having syntactical errors, logical errors with my understanding of the python language and its general logic and rules, or is there something simple here I’m missing? I am running my tests on a batch on 127 files, each range from 10,000 to 50,000 rows & 26 to 50 columns. But they each have ‘TD’ in cell A1, so this leads me to assume I’m having a logical misunderstanding here, because each and every file is being moved to the badPath.

error1

Asked By: Kyle Lucas

||

Source

Answer 1

I think you are closest in your second snippet of code because the move() is inside the if. However, the if is inside the open() context. You just need to move the if outside the context:

data = []

for f in TD_files:    
    with open(f, newline='', encoding='utf-8') as g: 
        r = csv.reader((line.replace('', '') for line in g))
        data = [line for line in r] 
        # data now has file contents having exhausted g

    # outside the context, the file is closed
    if data[0] != 'TD': 
        print('Not A Trend File') 
        shutil.move(os.path.join(path,f),badPath)

Answered By: quamrana

Answer 2

This produces a PermissionError because the processes are all tied up in one another.

We cannot move the file until it is closed; and it is open for the duration of the with block. However, with does not create a new scope in Python (very few things do), so we can easily compute values and then use them afterwards.

Rather than read the entire file in order to use indexing to get the first line, we can just read the first line out of the csv.reader, check whether the file is "bad", and set a flag to use afterwards:

for f in TD_files:    
    with open(f, newline='', encoding='utf-8') as g: 
        r = csv.reader((line.replace('', '') for line in g))
        is_trend = next(r)[0] == 'TD'
    if not is_trend:
        print('Not A Trend File')
        shutil.move(os.path.join(path, f), badPath)

(For that matter, we can probably just check whether the actual text of the file starts with TD, rather than parsing it as a CSV…)

Answered By: Karl Knechtel

Python conditional loop evaluating backwards

Question:

Answers: