Python threading Timer not filling output file

Question

I am using the module threading class Timer. I want to set up a process that will be repeated every day, generating an output json file every time is launched (day).

The problem is that it generates a file that is NOT filled until the whole process finishes (so if it has to run for an entire year, I shall wait all the year).

Here the code.

from processing import *
import sys
from datetime import datetime, timedelta
from threading import Timer

if __name__ == "__main__":
    '''
    '''
    #Function that generates a json file, that (works), but only is filled if Timer is NOT runing
    geojson_gen(sys.argv[1],
                sys.argv[2],
                sys.argv[3],
                out_filename = 'test_country'
                )

    for rep in range(10000):
        #Get the number of seconds for the next time Timer will launch the func. geojson_gen(). Here some code to get the number of seconds in which it shall be launched the timer
        next_date = (launch_date - start_date).seconds

        t = Timer(next_date , geojson_gen(sys.argv[1],
                                    sys.argv[2],
                                    sys.argv[3],
                                    out_filename = 'test_country'
                                    )
                 )
        t.start()

So all the script runs properly and if I comment all the Timer part, I get the json file. But when I start the process to run everyday, it generates empty json files (not filled).

What is wrong? How to get the json filled after the function geojson_gen() finished (and not after the whole Timer process)?

Thanks a lot!

Asked By: joueswant

||

Source

Answer 1

With the code given I don’t know why empty files are generated, but starting lots of Timers in parallel is not a good solution for your goal. Because what you want is not a parallel process, but a repeated, sequential one.

A more efficient way is to create a time-based loop. This example shows the basic idea:

import time

#set the wait time (a day in seconds)
waitTime = 60*60*24 

# set initial value
startTime = time.time()

# runs forever
while True:
    # check how much time has passed
    timeDiff = time.time() - startTime
    # if a day has passed, generate a json and update starttime
    if timeDiff > waitTime:
        geojson_gen()
        startTime = time.time()

This means you have to leave the script running though. An even cleaner solution is to only use the script that creates the json, then schedule in your OS to run that script daily. See this answer for a Windows example.

Answered By: J.D.

Answer 2

I had a similar issue with a script that uses threading.Timer(), the script was simply doing print().
This would work from the terminal, but when piping the output to a file, that file would only get updated very rarely. This then meant I couldn’t do tail -f.
This was solved by adding sys.stdout.flush().
Not sure if this helps the OP, but it may be a hint in the right direction.

Answered By: Dick Visser

Python threading Timer not filling output file

Question:

Answers: