Handling Connection Error when using Tweepy

Question

Hope you’re all well.

I’m trying to extract tweets continuously for a long period of time, using tweepy library. For that I have a python script with an infinite loop running. The only problem is that everytime the connection fails, for any possible reason, my script will crash and I will have to restart it. So I wanted to do it robust specially to connection errors,and for that I am using try…except condition but it doesn’t seem to work. I already tried everything I found online and nothing worked, can someone help me?

Here is my code:

consumer_key = ""
consumer_secret = ""
access_token = ""
access_token_secret = ""

auth = tweepy.OAuth1UserHandler(
   consumer_key, consumer_secret,
   access_token, access_token_secret
)
api = tweepy.API(auth)

city = 'LX'
country = 'PT'
coord = '38.736946,-9.142685'
radius = '25km'

while True:
    today = datetime.now()

    if len(str(today.month)) == 1:
        month = str(0) + str(today.month)
    else:
        month = str(today.month)    
    n=10
    tweets=tweepy.Cursor(api.search_tweets,q="",geocode=coord+','+radius,tweet_mode="extended").items(n)
 
    fname = f'{country}_{city}_{today.day}{month}{str(today.year)[2:]}.json'

    with open(fname, 'a') as outfile:
        for tweet in tweets:
            try: 
                outfile.write(json.dumps(tweet._json))
                outfile.write('n')
            except:
                os.system(f'echo error >> log_file.log')
                print('error')
                pass
        outfile.close()
    time.sleep(10)

(this is a sample code, so it’s possible it has some typos)
In the except clause I already tried with tweepy.TweepyException but without result too. To simulate a connection error I turn off my internet while running the script but I keep receiving an error and the script crashes, instead of just giving my the except block and continue running

Asked By: Filipa

||

Source

Answer 1

I used to run tweepy for months at a time. This is my brute force approach to the problem:
create a file called forever.py or whatever you want and use popen

#!/usr/bin/python3
from subprocess import Popen
import sys

filename = sys.argv[1]
arg1 = sys.argv[2]
while True:
    print("nStarting " + filename)
    p = Popen("python3 " + filename + ' ' + arg1, shell=True)
    p.wait()

Then from the command line :

python3 forever.py yourpythonscript.py arg1

Answered By: 1extralime

Handling Connection Error when using Tweepy

Question:

Answers: