using python requests to speed up for loop

Question

I have a python code and I want to speed it up using threads but when I try to I get the same lines getting duplicated, is there is any way I could speed it up without getting duplicate lines

code

import requests
import json

f = open("urls.json")
data = json.load(f)
def urls():
    for i in data['urls']:
        r = requests.get("https://" + i)
        print(r.headers)

Asked By: user19966157

||

Source

Answer 1

Make async or threaded calls.

So, you would do something like this:

import aiohttp
import asyncio
import time

start_time = time.time()


async def main():

    async with aiohttp.ClientSession() as session:

        for number in range(1, 151):
            pokemon_url = f'https://pokeapi.co/api/v2/pokemon/{number}'
            async with session.get(pokemon_url) as resp:
                pokemon = await resp.json()
                print(pokemon['name'])

asyncio.run(main())

Could also do multiprocessing as per the comment, but async is better for i/o type tasks.

Answered By: D.L

Answer 2

You can use ThreadPoolExecutor class from concurrent.futures. It is efficient way according to Thread class.

You can change the max_workers value according to your task

Here is the piece of code:

import requests
from concurrent.futures import ThreadPoolExecutor
import json

with open("urls.json") as f:
    data = json.load(f)

def urls():
    urls = ["https://" + url for url in data['urls']]
    print(urls)

    with ThreadPoolExecutor(max_workers=5) as pool:
        iterator = pool.map(requests.get,urls)

    for response in iterator:
        print(response.headers)
        print("n")

Answered By: Veysel Olgun

using python requests to speed up for loop

Question:

Answers: