threadpool

Making threads in a pool notice changes to global variables

Making threads in a pool notice changes to global variables Question: I’m running into an interesting circumstance with pools which I don’t fully understand. I was aware that if you edit any variable or object from a thread, changes will not be applied to the main thread and only exist in the isolated reality of …

Total answers: 1

"Data-loss while decompressing corrupted data" python threading

"Data-loss while decompressing corrupted data" python threading Question: I built a script to covert pdf files to excel and it works well. Now I want to use threading to make it faster but i keep getting this message "Data-loss while decompressing corrupted data" on the console and the output of my "extract" function are some …

Total answers: 1

Python ThreadPoolExecutor (concurrent.futures) memory leak

Python ThreadPoolExecutor (concurrent.futures) memory leak Question: Hello I’m trying to load a big list==list.txt and send it to Function==Do_something() with concurrent.futures.ThreadPoolExecutor The problem is that whatever I do, the memory gets heavy, At first I thought the reason is that i open list.txt into a variable as (list) and because of that i changed code …

Total answers: 1

Terminate all running threads at specific time

Terminate all running threads at specific time Question: i made a downloader based on a scheduler. So I wanted to download files at starting at a time of and stop before 6 am. I managed to achieve this using apscheduler that starts at 12 am, but i am not sure how to terminate all the …

Total answers: 1

Get thread index from a multiprocessing.pool.ThreadPool in Python

Get thread index from a multiprocessing.pool.ThreadPool in Python Question: I’m using a library whose multiprocessing is implemented using multiprocessing.pool.ThreadPool(processes). How is it possible to get/compute the thread index from the pool (starting from 0 to processes-1) ? I’ve been reading the documentation and searching the web without finding convincing solution. I can get the thread …

Total answers: 2

Python Postgres psycopg2 ThreadedConnectionPool exhausted

Python Postgres psycopg2 ThreadedConnectionPool exhausted Question: I have looked into several ‘too many clients’ related topic here but still can’t solve my problem, so I have to ask this again, for me specific case. Basically, I set up my local Postgres server and need to do tens of thousands of queries, so I used the …

Total answers: 4

What's the difference between ThreadPool vs Pool in the multiprocessing module?

What's the difference between ThreadPool vs Pool in the multiprocessing module? Question: Whats the difference between ThreadPool and Pool in multiprocessing module. When I try my code out, this is the main difference I see: from multiprocessing import Pool import os, time print(“hi outside of main()”) def hello(x): print(“inside hello()”) print(“Proccess id: “, os.getpid()) time.sleep(3) …

Total answers: 1

Python 3: does Pool keep the original order of data passed to map?

Python 3: does Pool keep the original order of data passed to map? Question: I have written a little script to distribute workload between 4 threads and to test whether the results stay ordered (in respect to the order of the input): from multiprocessing import Pool import numpy as np import time import random rows …

Total answers: 3

Controlling scheduling priority of python threads?

Controlling scheduling priority of python threads? Question: I’ve written a script that uses two thread pools of ten threads each to pull in data from an API. The thread pool implements this code on ActiveState. Each thread pool is monitoring a Redis database via PubSub for new entries. When a new entry is published, python …

Total answers: 5