Python Memory Error
Question:
I’m runing into Memory Error
Using python 2.6, pyodbc
The code loops through several Sql statements. Singleton run Ok, but the loops get stuck.
Sometimes Loop works ok.
for Sql in LoopList:
f = csr.execute(Sql)
LL = list(f)
The second element in loop crashes.
Sql:
Sql =""" SELECT * FROM group WHERE
C>10 AND M <0 AND S<0
AND TC >= 200 AND OC >=1000 and Penny =.01
ORDER BY MSlp
"""
Answers:
Your query is returning too much data to fit into memory. The execute is probably returning an iterator which only needs to keep one item in memory at a time, but converting that into a list requires every single item returned from the query to fit into memory.
The solution was to do sql looplist inside MySQL as Stored procedure, call from python. This eliminated the python error.
also use numpy for testing bytesize
Since your query is returning too much data that it cannot be stored in RAM, for a quick solution try increasing your virtual memory.
- Open My Computer
- Right click and select Properties
- Go to Advanced System Settings
- Click on Advance Tab
- Click on settings under Performance
- Click on Change under Advance Tab
- Increase the memory size, that will increase virtual memory size.
It will convert some of your hard disk space to virtual memory, and your issue will be resolved now if your data fits into allocated memory.
I’m runing into Memory Error
Using python 2.6, pyodbc
The code loops through several Sql statements. Singleton run Ok, but the loops get stuck.
Sometimes Loop works ok.
for Sql in LoopList:
f = csr.execute(Sql)
LL = list(f)
The second element in loop crashes.
Sql:
Sql =""" SELECT * FROM group WHERE
C>10 AND M <0 AND S<0
AND TC >= 200 AND OC >=1000 and Penny =.01
ORDER BY MSlp
"""
Your query is returning too much data to fit into memory. The execute is probably returning an iterator which only needs to keep one item in memory at a time, but converting that into a list requires every single item returned from the query to fit into memory.
The solution was to do sql looplist inside MySQL as Stored procedure, call from python. This eliminated the python error.
also use numpy for testing bytesize
Since your query is returning too much data that it cannot be stored in RAM, for a quick solution try increasing your virtual memory.
- Open My Computer
- Right click and select Properties
- Go to Advanced System Settings
- Click on Advance Tab
- Click on settings under Performance
- Click on Change under Advance Tab
- Increase the memory size, that will increase virtual memory size.
It will convert some of your hard disk space to virtual memory, and your issue will be resolved now if your data fits into allocated memory.