What is a good size (in bytes) for a log file?
Question:
I am using the python logging modules RotatingFileHandler, and you can set the maximum size of each log file. What is a good maximum size for a log file? Please give your answer in bytes.
Answers:
It completely depends on the external variables of the system. For instance:
- Are you running on an embedded device whose only external storage is a 1MB SD card, or do you have full access to a 1TB hard drive?
- Are you logging each time you enter/exit a function, or are you only logging one or two caught exceptions throughout the whole system?
- Is the purpose of these logs to be sent back to the developer for support? A 1kb log file isn’t going to help you much, but you probably don’t need 200MB of logs for a single support issue.
Without these kinds of details, there is no good answer to your question (and there might not be a good answer even with these details).
Size isn’t as important to me as dividing at sensible chronological points. I prefer one log file per day, however, if the file won’t open with any notepad program you have at your disposal, it is too big and you might want to go with hourly logs.
As the other answers have said, there is a no hard and fast answer. It depends so much on your app and your environment. Here’s some guidelines I use.
For a multi-user app on a typical server:
Configure your logging to generate no more than 1 or 2 entries per user action for production, and then rotate it daily. Keep as many days as you have disk space for, or your data retention/privacy policies allow for. If you want auditing, you probably want a separate solution.
For a single-user app:
Try and keep enough information to diagnose anything weird that might happen. No more than 2 or 3 entries per user action though, unless you are doing batch operations. Don’t put more than 2MB in a file, so the user can email it you. Don’t keep more than 50MB of logs, because it’s probably not your space you are wasting here.
My default logging setup:
RotatingFileHandler(filename, maxBytes=10*1024*1024, backupCount=5)
I am using the python logging modules RotatingFileHandler, and you can set the maximum size of each log file. What is a good maximum size for a log file? Please give your answer in bytes.
It completely depends on the external variables of the system. For instance:
- Are you running on an embedded device whose only external storage is a 1MB SD card, or do you have full access to a 1TB hard drive?
- Are you logging each time you enter/exit a function, or are you only logging one or two caught exceptions throughout the whole system?
- Is the purpose of these logs to be sent back to the developer for support? A 1kb log file isn’t going to help you much, but you probably don’t need 200MB of logs for a single support issue.
Without these kinds of details, there is no good answer to your question (and there might not be a good answer even with these details).
Size isn’t as important to me as dividing at sensible chronological points. I prefer one log file per day, however, if the file won’t open with any notepad program you have at your disposal, it is too big and you might want to go with hourly logs.
As the other answers have said, there is a no hard and fast answer. It depends so much on your app and your environment. Here’s some guidelines I use.
For a multi-user app on a typical server:
Configure your logging to generate no more than 1 or 2 entries per user action for production, and then rotate it daily. Keep as many days as you have disk space for, or your data retention/privacy policies allow for. If you want auditing, you probably want a separate solution.
For a single-user app:
Try and keep enough information to diagnose anything weird that might happen. No more than 2 or 3 entries per user action though, unless you are doing batch operations. Don’t put more than 2MB in a file, so the user can email it you. Don’t keep more than 50MB of logs, because it’s probably not your space you are wasting here.
My default logging setup:
RotatingFileHandler(filename, maxBytes=10*1024*1024, backupCount=5)