Finding an efficient way to distribute elements of a list between multiple processes in python Question: Suppose I have a list of strings like so: a = [‘string1’, ‘string2’, …, ‘stringN’] Was wondering what could be the most efficient way to make use of "8" processes using these "N" strings in the said list as …
hog+svm classification takes too mcuh time Question: i have directory called Animal- which consist of two subdirectory(cats and dogs), create a full path for each file does not take too much time and it returns quickly, here is sample code : import cv2 import numpy as np from skimage.feature import hog from sklearn.svm import SVC …
Which multiprocessing method map or apply_async? Question: I have a function: def movingWinStretch(u0,u1): # u0,u1 are 1D arrays # do a bunch of stuff to u0 and u1 return C , epsArray, tSamp When I do this workflow on smaller amounts of data I use a couple of nested for loops to loop through the …
Parallel while loop with unknown number of calls Question: I have written a function that does some calculations, and that everytime it is called it returns a different result, since it uses a different seed for the random number generator. In general, I want to run this functions many times, in order to obtain many …
parallelization of downloading thousands of files using wget Question: I have thousands of the files like below to be downloaded. urls = [‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0450.061.2019001110251.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0455.061.2019001110452.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0500.061.2019001110658.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0535.061.2019001110116.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0555.061.2019001132709.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0615.061.2019001132734.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0630.061.2019001132950.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0635.061.2019001133203.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0650.061.2019001132727.hdf’, ‘https://ladsweb.modaps.eosdis.nasa.gov//archive/allData/61/MOD03/2019/001/MOD03.A2019001.0655.061.2019001132653.hdf’] I can download them one by one using wget as follows. #wget is here, https://eternallybored.org/misc/wget/1.21.3/64/wget.exe import os, glob, subprocess import itertools import multiprocessing …
Parallelization of un-bzipping millions of files Question: I have millions of compressed .bz2 files which I need to uncompressed. Can uncompression be parallelized ? I have access to the server with many cpu cores for the purpose. I worked with the following code which is correct but it is extremely slow. import os, glob, bz2 …
Parallelize loop with "non-mapable" function in Python Question: I solve numerically integral equation on the time interval from 0 to Tmax with predefined step dT. I do it in the for loop: list_of_values = [] for i in range(dT,Tmax+dT,dT): func_at_t= my_fancy_solver(func_at_t) list_of_values.append(func_at_t) The function my_fancy_solver has one argument and inside it looks as follows: def …
Nested vmap in pmap – JAX Question: I currently can run simulations in parallel on one GPU using vmap. To speed things up, I want to batch the simulations over multiple GPU devices using pmap. However, when pmapping the vmapped function I get a tracing error. The code I use to get a trajectory state …
Python Parallel Processing with ThreadPoolExecutor Gives Wrong Results with Keras Model Question: I am using parallel processing using the concurrent.futures.ThreadPoolExecutor class to make multiple predictions using a Keras model for different sets of weights. But the Keras model predictions using parallel processing are not correct. This is a reproducible sample code that creates 10 sets …
Multiprocessing not using whole CPU Question: I’m testing python’s module "multiprocessing". I’m trying to compute pi using a montecarlo technique using my 12 threads ryzen 5 5600. The problem is that my cpu is not fully used, instead only 47% is used. I leave you my code below, changing the value of n_cpu leads to …
