keep data in memory persistently
Question:
I want to write a Python script which loads 2 GB of data from the hard disk into memory and then whenever requested by other program, it must get an input and do some calculations on this data based on the input. the important thing for me is to keep this 2 GB data in memory persistently to speed up the calculations and more importantly avoid huge I/O load.
how should I keep the data in memory forever? or more generally, how should I solve such problem in Python?
Answers:
Depending on what kind of data you have, you can keep the data in a Python list, set, hashmap or any other data structure. If this is just meant to be a cache, you can use a server like Redis or memcached too.
There is nothing special about loading data to memory "forever" or doing it every time you need it. You can just load into Python variables and keep them around.
Make sure you have 2GB of free and available RAM and then use the mmap module (https://docs.python.org/3/library/mmap.html) to map the entire array into active memory.
You can package the function that loads the data, then use the decorator ‘cache’ or ‘lru_cache’ from functools (https://docs.python.org/3/library/functools.html) to memorize the function that was created.
I want to write a Python script which loads 2 GB of data from the hard disk into memory and then whenever requested by other program, it must get an input and do some calculations on this data based on the input. the important thing for me is to keep this 2 GB data in memory persistently to speed up the calculations and more importantly avoid huge I/O load.
how should I keep the data in memory forever? or more generally, how should I solve such problem in Python?
Depending on what kind of data you have, you can keep the data in a Python list, set, hashmap or any other data structure. If this is just meant to be a cache, you can use a server like Redis or memcached too.
There is nothing special about loading data to memory "forever" or doing it every time you need it. You can just load into Python variables and keep them around.
Make sure you have 2GB of free and available RAM and then use the mmap module (https://docs.python.org/3/library/mmap.html) to map the entire array into active memory.
You can package the function that loads the data, then use the decorator ‘cache’ or ‘lru_cache’ from functools (https://docs.python.org/3/library/functools.html) to memorize the function that was created.