Maximum size of object that can be saved in memcached
Question:
I need to return a rather big file (11MB) to the user. For certain reasons, I can’t just provide a direct url to the file (http://www.sample.com/mybigfile.exe); instead it must be accessed through code.
Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know). Everything seems to work, fine (no errors), but when I try to retrieve the file from memcached I always get None, as if the file wasn’t cached.
Is there a size limit to what can be saved?
Here’s the code:
def download_demo():
"""
Returns the demo file
"""
KEY = "xyz"
TIME = 86400 #24 hours
buff = memc.get(KEY)
if not buff:
file = open(FILENAME, 'r')
buff = file.read()
memc.set(KEY, buff, TIME)
print "Content-Type:application/x-downloadnContent-Disposition:attachment;filename=%snContent-Length:%snn%s" % (os.path.split(FILENAME)[-1], len(buff), buff)
Answers:
The maximum data size is 1mb per item
Update: This is a 2009 answer. And in that date the info was accurate and the source was official. Now in 2014 memcached can store 128mb instead of 1mb (but the developers didn’t bother to update the official FAQ). The only reference someone could find id an obscure page that probably will be dead in one year.
There are two entries about that in the memcached FAQ :
The answer to the first one is (quoting, emphasis mine) :
The maximum size of a value you can
store in memcached is 1 megabyte. If
your data is larger, consider
clientside compression or splitting
the value up into multiple keys.
So I’m guessing your 11MB file is quite too big to fit in one memcached entry.
Increasing the size of an object is possible, as per other answers.
As of memcache 1.4.2, this is a user-configurable parameter: ReleaseNotes142 * memcached @ Github
Configurable maximum item size
Many people have asked for memcached to be able to store items larger
than 1MB, while it’s generally recommended that one not do this, it is
now supported on the commandline.
A few enlightened folk have also asked for memcached to reduce the
maximum item size. That is also an option.
The new -I parameter allows you to specify the maximum item size at
runtime. It supports a unit postfix to allow for natural expression of
item size.
Examples:
memcached -I 128k # Refuse items larger than 128k.
memcached -I 10m # Allow objects up to 10MB
To summarise the necessary steps:
1) Update Memcache to version 1.4.2 or later.
2) Add the flag -I 15M (or however many megabytes) to your memcache run command.
That’s either command line, or in Ubuntu, add the line
-I 15M
to anywhere in /etc/memcached.conf and restart the service.
3) Add the necessary flag to the client in memcache.
import memcache
memc = memcache.Client(['localhost'], server_max_value_length=1024*1024*15)
memc.set(KEY, buff, TIME)
If you don’t have direct access to the memcache client (i.e. working through a framework), then just hack the memcache code directly.
On Ubuntu, it’s /usr/local/lib/python2.7/dist-packages/memcache.py. Change the line:
SERVER_MAX_ITEM_LENGTH = 1024 * 1024
to
SERVER_MAX_ITEM_LENGTH = 1024 * 1024 * 15
Obviously, you’ll need to do this hack over again if you update memcache, but it’s a very simple and quick fix.
Following article explained why it limit max size of a single file to 1M.
http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/
Basicly, there are small pieces of memory pages in memcache, inside which objects reside.
And the default page size seems to be 1M, so the max object a page can contain is limited to 1M.
Even though you can config it to increase the size, but I think this might have some kind of performance trade-off.
If you are using python memcached client, please also make sure to increase the limit in client memcache.py file along with the memcached server.
~/anaconda/lib/python2.7/site-packages/memcache.py
SERVER_MAX_KEY_LENGTH = 250
SERVER_MAX_VALUE_LENGTH = 1024 * 1024 * 15
you can see in memcached console ( telnet localhost 11211 )
stats slabs
STAT 48:chunk_size 3677344
Nobody seems to address this part of the question:
Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know).
This is not a good idea. All modern file systems have efficient caching mechanisms. Reading a file from disk that was read a few seconds earlier will typically still be in memory cache. This is much faster than accessing memcached over a network.
If you are running memcached on 127.0.0.1, the memory it claimed will only be used for that single file, wereas if you simply rely on the caching of your operating system the memory can be used for various purposes depending on the jobs at hand.
2022 Update –
For open source Memcached,
- the minimum value for max-item-size(-I, –max-item-size) is 1KiB
- the maximum value is 1 GiB
- the default value is 1 MiB
if (unit == 'k' || unit == 'K')
size_max *= 1024;
if (unit == 'm' || unit == 'M')
size_max *= 1024 * 1024;
if (settings.item_size_max > (ITEM_SIZE_MAX_UPPER_LIMIT)) {
fprintf(stderr, "Cannot set item size limit higher than a gigabyte.n");
exit(EX_USAGE);
}
In case of cloud memcache variant this value will differ – For example here is the limit on google cloud memcache – gcloud docs
Note: If you are planning to increase the max-item-size value then inspect/increase the amount of memory allocated for the cache(maxbytes
: this is the overall Memcached memory). use the -m option to specify the amount of RAM to be allocated (in megabytes). Default is 64MB.
Both settings are available to inspect in memcached statistics (size in bytes):
stats settings
STAT maxbytes 67108864
STAT item_size_max 1048576
I need to return a rather big file (11MB) to the user. For certain reasons, I can’t just provide a direct url to the file (http://www.sample.com/mybigfile.exe); instead it must be accessed through code.
Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know). Everything seems to work, fine (no errors), but when I try to retrieve the file from memcached I always get None, as if the file wasn’t cached.
Is there a size limit to what can be saved?
Here’s the code:
def download_demo():
"""
Returns the demo file
"""
KEY = "xyz"
TIME = 86400 #24 hours
buff = memc.get(KEY)
if not buff:
file = open(FILENAME, 'r')
buff = file.read()
memc.set(KEY, buff, TIME)
print "Content-Type:application/x-downloadnContent-Disposition:attachment;filename=%snContent-Length:%snn%s" % (os.path.split(FILENAME)[-1], len(buff), buff)
The maximum data size is 1mb per item
Update: This is a 2009 answer. And in that date the info was accurate and the source was official. Now in 2014 memcached can store 128mb instead of 1mb (but the developers didn’t bother to update the official FAQ). The only reference someone could find id an obscure page that probably will be dead in one year.
There are two entries about that in the memcached FAQ :
The answer to the first one is (quoting, emphasis mine) :
The maximum size of a value you can
store in memcached is 1 megabyte. If
your data is larger, consider
clientside compression or splitting
the value up into multiple keys.
So I’m guessing your 11MB file is quite too big to fit in one memcached entry.
Increasing the size of an object is possible, as per other answers.
As of memcache 1.4.2, this is a user-configurable parameter: ReleaseNotes142 * memcached @ Github
Configurable maximum item size
Many people have asked for memcached to be able to store items larger
than 1MB, while it’s generally recommended that one not do this, it is
now supported on the commandline.A few enlightened folk have also asked for memcached to reduce the
maximum item size. That is also an option.The new -I parameter allows you to specify the maximum item size at
runtime. It supports a unit postfix to allow for natural expression of
item size.Examples:
memcached -I 128k # Refuse items larger than 128k. memcached -I 10m # Allow objects up to 10MB
To summarise the necessary steps:
1) Update Memcache to version 1.4.2 or later.
2) Add the flag -I 15M (or however many megabytes) to your memcache run command.
That’s either command line, or in Ubuntu, add the line
-I 15M
to anywhere in /etc/memcached.conf and restart the service.
3) Add the necessary flag to the client in memcache.
import memcache
memc = memcache.Client(['localhost'], server_max_value_length=1024*1024*15)
memc.set(KEY, buff, TIME)
If you don’t have direct access to the memcache client (i.e. working through a framework), then just hack the memcache code directly.
On Ubuntu, it’s /usr/local/lib/python2.7/dist-packages/memcache.py. Change the line:
SERVER_MAX_ITEM_LENGTH = 1024 * 1024
to
SERVER_MAX_ITEM_LENGTH = 1024 * 1024 * 15
Obviously, you’ll need to do this hack over again if you update memcache, but it’s a very simple and quick fix.
Following article explained why it limit max size of a single file to 1M.
http://www.mikeperham.com/2009/06/22/slabs-pages-chunks-and-memcached/
Basicly, there are small pieces of memory pages in memcache, inside which objects reside.
And the default page size seems to be 1M, so the max object a page can contain is limited to 1M.
Even though you can config it to increase the size, but I think this might have some kind of performance trade-off.
If you are using python memcached client, please also make sure to increase the limit in client memcache.py file along with the memcached server.
~/anaconda/lib/python2.7/site-packages/memcache.py
SERVER_MAX_KEY_LENGTH = 250
SERVER_MAX_VALUE_LENGTH = 1024 * 1024 * 15
you can see in memcached console ( telnet localhost 11211 )
stats slabs
STAT 48:chunk_size 3677344
Nobody seems to address this part of the question:
Instead of having to read it from disk over and over, I thought of saving it in memcached (if this is not a good idea, let me know).
This is not a good idea. All modern file systems have efficient caching mechanisms. Reading a file from disk that was read a few seconds earlier will typically still be in memory cache. This is much faster than accessing memcached over a network.
If you are running memcached on 127.0.0.1, the memory it claimed will only be used for that single file, wereas if you simply rely on the caching of your operating system the memory can be used for various purposes depending on the jobs at hand.
2022 Update –
For open source Memcached,
- the minimum value for max-item-size(-I, –max-item-size) is 1KiB
- the maximum value is 1 GiB
- the default value is 1 MiB
if (unit == 'k' || unit == 'K')
size_max *= 1024;
if (unit == 'm' || unit == 'M')
size_max *= 1024 * 1024;
if (settings.item_size_max > (ITEM_SIZE_MAX_UPPER_LIMIT)) {
fprintf(stderr, "Cannot set item size limit higher than a gigabyte.n");
exit(EX_USAGE);
}
In case of cloud memcache variant this value will differ – For example here is the limit on google cloud memcache – gcloud docs
Note: If you are planning to increase the max-item-size value then inspect/increase the amount of memory allocated for the cache(maxbytes
: this is the overall Memcached memory). use the -m option to specify the amount of RAM to be allocated (in megabytes). Default is 64MB.
Both settings are available to inspect in memcached statistics (size in bytes):
stats settings
STAT maxbytes 67108864
STAT item_size_max 1048576