How to prevent the error "unable to allocate x.xx GiB for an array with shape (xx, xxx) and data type xxxxx" without restarting VS Code?
Question:
I am working with a lot of data in a jupyter notebook. From my research I understand that the memory (RAM/SWAP) is not sufficient to (temporarily) store my large dataframe which causes the code to exit with the error unable to allocate x.xx GiB for an array with shape (xx, xxx) and data type xxxxx.
However, if I then completely close VS Code or restart the kernel of my jupyter notebook, I do not get this error. I only get this error after I have been working/coding in jupyter notebook for a long time. I tried del <variable names>, but that does not actually free the memory. I then also tried gc.collect() but that still leads to the error.
I mainly need to run two cells in jupyter notebook. The first imports millions of rows out of excel. This cell takes about 1 hour. The second cell cleans the data. I want to keep my excel imports, because those take really long. Everything else can be deleted from memory and free up space such that the error does not occur without having to restart the kernel.
Answers:
For future reference, I used pickling to solve this issue.
I am working with a lot of data in a jupyter notebook. From my research I understand that the memory (RAM/SWAP) is not sufficient to (temporarily) store my large dataframe which causes the code to exit with the error unable to allocate x.xx GiB for an array with shape (xx, xxx) and data type xxxxx.
However, if I then completely close VS Code or restart the kernel of my jupyter notebook, I do not get this error. I only get this error after I have been working/coding in jupyter notebook for a long time. I tried del <variable names>, but that does not actually free the memory. I then also tried gc.collect() but that still leads to the error.
I mainly need to run two cells in jupyter notebook. The first imports millions of rows out of excel. This cell takes about 1 hour. The second cell cleans the data. I want to keep my excel imports, because those take really long. Everything else can be deleted from memory and free up space such that the error does not occur without having to restart the kernel.
For future reference, I used pickling to solve this issue.