Django, StreamingHTTPResponse and JSON

Question

What is the correct way to stream JSON response from Django? E.g. use ‘StreamingHttpResponse` for the following:

def sample_json_view(self):

    data = { ... }

    resp = http.HttpResponse(content_type="application/json")
    json.dump(data, resp)
    return resp

My goal is to use StreamingHttpResponse to minimize the latency in the view. Does StreamingHttpResponse offer benefit over HttpResponse where we are writing?

Or: Does writing to file-like HttpResponse object (like with json.dump) make Django / mod_wsgi to buffer the whole response on the server before starting to stream the response to the client? (increasing latency, all JSON response must be generated first).

Asked By: Mikko Ohtamaa

||

Source

Answer 1

This depends on how your data is being generated, and if you need the content rendered before all of the data is generated. The Django docs seem to discourage this, saying “StreamingHttpResponse should only be used in situations where it is absolutely required that the whole content isn’t iterated before transferring the data to the client.”

For an example of how to correctly use StreamingHttpResponse, see Django 1.5 – using the new StreamingHttpResponse

Answered By: Aaron H

Answer 2

hopefully you found your solution. But for the people that are still getting forwarded to this question, this is how I stream a JsonResponse.

def get_data(request):
    query = model.objects.all()
    
    def data_stream():
        for element in query:
            element_dict = {} # This is where you serialize your data
            yield json.dumps(element_dict)

    return StreamingHttpResponse(data_stream())

With this, when you print each chunk of the stream you will get something like b'{element1: element1_value, element2: element2_value}' which you can serialice individually, as the stream is an iterable. This is quite efficient as you will only have a single value of the query on memory.

If you have more questions, I am open to answer them.

Answered By: jacobitosuperstar

Django, StreamingHTTPResponse and JSON

Question:

Answers: