Script suddenly using all RAM Question: I have a python script I am using to convert some very densely formatted csv files into another format that I need. The csv files are quite large (3GB) so I read them in chunks to avoid using all the RAM (I have 32GB of RAM on the machine …
Memory issue while running ARIMA model Question: I am trying to run my ARIMA model and am getting the below error:- MemoryError: Unable to allocate 52.4 GiB for an array with shape (83873, 83873) and data type float64 My python/anaconda is installed in the C drive and has somewhere around 110GB free space but still …
Allocation of Memory with python to pass to dll Question: I got a dll which expects a memory pointer to a C Type Byte Array. The dll will read and modify the Array and will also put some extra data at the end of the array. How do I allocate 1MB memory as C Type …
Dask Dataframe nunique operation: Worker running out of memory (MRE) Question: tl;dr I want to dd.read_parquet(‘*.parq’)[‘column’].nunique().compute() but I get WARNING – Worker exceeded 95% memory budget. Restarting a couple of times before the workers get killed altogether. Long version I have a dataset with 10 billion rows, ~20 columns, and a single machine with around …
Making Sieve of Eratosthenes more memory efficient in python? Question: Sieve of Eratosthenes memory constraint issue Im currently trying to implement a version of the sieve of eratosthenes for a Kattis problem, however, I am running into some memory constraints that my implementation wont pass. Here is a link to the problem statement. In short …
How do we approximately calculate how much memory is required to run a program? Question: Today I was trying to implement an object detection API in Tensorflow. After carrying out the training process, I was trying to run the program to detect objects in webcam. As I was running it, the following message was printed …
Numpy overflow in short scalars Question: I have a large .wav file array (200k samples) loaded in with scipy.io.wavfile. I tried to make a histogram of the data using matplotlib.pyplot hist with auto binning. It returned the error: ValueError: Number of samples, -72, must be non-negative. So I decided to set the bins myself using …
How to increase Jupyter notebook Memory limit? Question: I am using jupyter notebook with Python3 on windows 10. My computer has 8GB RAM and at least 4GB of my RAM is free. But when I want to make a numpy ndArray with size 6000*6000 with this command: np.zeros((6000, 6000), dtype=’float64′) I got this : Unable …
How to clear jupyter memory without restarting notebook Question: I am using a 3D Convolutional Neural Network for my thesis and I am trying to train the network with an input of 256×256 images, 22 channels, 5 pictures, using 8×8 sliding window with 90 degree rotation data augmentation. So the input size is (262144,22,8,8,5). The …
Why can a 352GB NumPy ndarray be used on an 8GB memory macOS computer? Question: import numpy as np array = np.zeros((210000, 210000)) # default numpy.float64 array.nbytes When I run the above code on my 8GB memory MacBook with macOS, no error occurs. But running the same code on a 16GB memory PC with Windows …
