Memory Error Python Numpy Array

Question

I’m running into a memory error when trying to evaluate a function at a number of points and store in an array. I’ve read a few other posts about similar issues and I don’t think that I am exceeding the memory limits of my system:

              total        used        free      shared  buff/cache   available
Mem:      131898384     8964068   122074628       15128      859688   121905868
Swap:      14648316     1001760    13646556

and also

MemTotal:       131898384 kB
MemFree:        75056308 kB
MemAvailable:   74889296 kB

The error message is:

PDF= np.zeros([T,Y])
MemoryError

My code is:

Y=40000
T = 200000
tmin = 0
timestep = 10
tmax = timestep*T
TD_Psi = np.zeros([T,Y],'complex')
t = np.linspace(tmin, tmax, T)

for j in range(T):
    for i in range(M):
        TD_Psi[j] = TD_Psi[j] + c[i]*MO_basis[i]*np.exp(-evals[i]*t[j]*1j)

PDF= np.zeros([T,Y])
for time in range(T):
    PDF[time] = np.real(np.conjugate(TD_Psi[time])*TD_Psi[time])

I’m not sure if the problem is in my code or my system and what I can do to resolve this issue. It runs up until T = 150000 before giving me the error.

Thank you very much for your help.

Asked By: XBB

||

Source

Answer 1

np.zeros([T,Y],'complex') is 128 GB. np.zeros([T,Y]) is another 64 GB. You don’t have 192 GB of RAM. You’re out of memory.

Answered By: user2357112

Answer 2

Since you’re running out of memory and the data cannot be stored in RAM, for a quick solution try increasing your virtual memory.

Open My Computer
Right click and select Properties
Go to Advanced System Settings
Click on Advance Tab
Click on settings under Performance
Click on Change under Advance Tab
Increase the memory size, that will increase virtual memory size.

Make sure to select as much as you need, it will convert your hard disk space to virtual memory, and your issue will be resolved now if your data fits into allocated memory.

Answered By: Asif Mehmood

Memory Error Python Numpy Array

Question:

Answers: