Django debug toolbar: how do I profile a file download?
Question:
My Django webapp lets users download text files that are generated on the fly:
response = HttpResponse(my_file_contents)
response['Content-Disposition'] = 'attachment; filename="my file.txt"'
return response
I installed Django Debug Toolbar (0.11.0, since I cannot get 1.0.1 to work), but when I click to make the download, the toolbar doesn’t show info about the file that was downloaded, presumably because that is a separate page/request (or maybe because it’s a non-HTML file). The downloaded file doesn’t contain any debug info either.
How can I profile the performance of this file download?
Answers:
You are right, this is one of the cases where the Debug Toolbar cannot help you. I would recommend using log files to time your request times. For instance, if you are using Nginx then you can use its syntax for adding extra information to log files. For instance the following line adds the response time for each request:
log_format timed_combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $gzip_ratio';
If you prefer a Django app solution, you can check out django-timelog.
An alternative, temporary solution if you are focused on profiling DB queries is to simply not return the file download response, and instead load a template (any valid Django template in your application should work). DDT still logs all the queries and you can see them on the subsequent page. This works because often what you are interested in is the queries that take place in order to build the data that is being prepared for download. The actual process of taking some data already in-hand and returning a response is usually very quick.
So let’s say you have a form where this download can be requested. Typically your view would return something like:
# (Do something here to collect data)
response = HttpResponse(export_data, content_type=content_type)
response['Content-Disposition'] = 'attachment; filename=somefile.txt'
return response
Just comment this out and return a regular rendered template instead — you don’t need to display the data if you don’t want to. If using a mixin like TemplateView or FormView, this might be simply commenting out the above and then Django will render the template as if the download action hadn’t been performed. Or, just render any Django template in your application. Now, open the DDT toolbar — there’s all your queries!
You can use django-debug-toolbar-force.
It allows you to force template rendering for any url by appending ?debug-toolbar.
This allows you to access the toolbar and all of its instrumentation during development for any view without any code changes.
Caveat: I am not sure how actively maintained it is as of now, the last commit is from 2020.
@timothyashaw’s answer makes sense, but it requires temporary code modifications.
Alternatively, we can write a piece of middleware that catches all "file" responses and returns a dummy response instead. Django-debug-toolbar conveniently provides the Panel class for this, which has the advantage that we can easily toggle our middleware on or off via the DjDT toolbar.
The following example is basically a modified version of the default "Intercept redirects" panel:
from debug_toolbar.panels import Panel
from django.http import HttpResponse
class FileInterceptsPanel(Panel):
has_content = False
title = 'Intercept files'
def process_request(self, request):
"""If the response contains a file, replace it by a dummy response"""
response = super().process_request(request)
response_is_file = ... # use whatever test suits your case
if response_is_file:
response = HttpResponse('<body>debug file response</body>')
return response
Once configured, you’ll see something like this in your toolbar:
Some notes:
-
The FileInterceptsPanel should be inserted into DEBUG_TOOLBAR_PANELS in your settings.py, as explained in the docs.
-
The FileInterceptsPanel will be turned on by default, but you can add it to DEBUG_TOOLBAR_CONFIG['DISABLE_PANELS'] in your settings.py to turn it off by default.
-
The response_is_file test depends on your specific use-case. In the OP’s example it could be something like 'attachment' in response.get('Content-Disposition', ''). Alternatively, we could e.g. test for instances like FileResponse, StreamingHttpResponse, JsonResponse, or test for response.streaming.
-
If you’re catching JsonResponse instances, beware that django-debug-toolbar also uses those for the detail panels. These could be excluded e.g. by testing for b'djdt' not in response.content.
My Django webapp lets users download text files that are generated on the fly:
response = HttpResponse(my_file_contents)
response['Content-Disposition'] = 'attachment; filename="my file.txt"'
return response
I installed Django Debug Toolbar (0.11.0, since I cannot get 1.0.1 to work), but when I click to make the download, the toolbar doesn’t show info about the file that was downloaded, presumably because that is a separate page/request (or maybe because it’s a non-HTML file). The downloaded file doesn’t contain any debug info either.
How can I profile the performance of this file download?
You are right, this is one of the cases where the Debug Toolbar cannot help you. I would recommend using log files to time your request times. For instance, if you are using Nginx then you can use its syntax for adding extra information to log files. For instance the following line adds the response time for each request:
log_format timed_combined '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'$request_time $upstream_response_time $gzip_ratio';
If you prefer a Django app solution, you can check out django-timelog.
An alternative, temporary solution if you are focused on profiling DB queries is to simply not return the file download response, and instead load a template (any valid Django template in your application should work). DDT still logs all the queries and you can see them on the subsequent page. This works because often what you are interested in is the queries that take place in order to build the data that is being prepared for download. The actual process of taking some data already in-hand and returning a response is usually very quick.
So let’s say you have a form where this download can be requested. Typically your view would return something like:
# (Do something here to collect data)
response = HttpResponse(export_data, content_type=content_type)
response['Content-Disposition'] = 'attachment; filename=somefile.txt'
return response
Just comment this out and return a regular rendered template instead — you don’t need to display the data if you don’t want to. If using a mixin like TemplateView or FormView, this might be simply commenting out the above and then Django will render the template as if the download action hadn’t been performed. Or, just render any Django template in your application. Now, open the DDT toolbar — there’s all your queries!
You can use django-debug-toolbar-force.
It allows you to force template rendering for any url by appending ?debug-toolbar.
This allows you to access the toolbar and all of its instrumentation during development for any view without any code changes.
Caveat: I am not sure how actively maintained it is as of now, the last commit is from 2020.
@timothyashaw’s answer makes sense, but it requires temporary code modifications.
Alternatively, we can write a piece of middleware that catches all "file" responses and returns a dummy response instead. Django-debug-toolbar conveniently provides the Panel class for this, which has the advantage that we can easily toggle our middleware on or off via the DjDT toolbar.
The following example is basically a modified version of the default "Intercept redirects" panel:
from debug_toolbar.panels import Panel
from django.http import HttpResponse
class FileInterceptsPanel(Panel):
has_content = False
title = 'Intercept files'
def process_request(self, request):
"""If the response contains a file, replace it by a dummy response"""
response = super().process_request(request)
response_is_file = ... # use whatever test suits your case
if response_is_file:
response = HttpResponse('<body>debug file response</body>')
return response
Once configured, you’ll see something like this in your toolbar:
Some notes:
-
The
FileInterceptsPanelshould be inserted intoDEBUG_TOOLBAR_PANELSin yoursettings.py, as explained in the docs. -
The
FileInterceptsPanelwill be turned on by default, but you can add it toDEBUG_TOOLBAR_CONFIG['DISABLE_PANELS']in yoursettings.pyto turn it off by default. -
The
response_is_filetest depends on your specific use-case. In the OP’s example it could be something like'attachment' in response.get('Content-Disposition', ''). Alternatively, we could e.g. test for instances like FileResponse, StreamingHttpResponse, JsonResponse, or test forresponse.streaming. -
If you’re catching
JsonResponseinstances, beware that django-debug-toolbar also uses those for the detail panels. These could be excluded e.g. by testing forb'djdt' not in response.content.