Django long request timeout

Question

I have a view in a Django application that take as input a large CSV file. It cycle all the rows and insert datas on DB.
Locally I have no problem but when I deploy my proj online the view give me back a timeout after some time.
My webserver configuration is Nginx + Gunicorn.
I tried to increase timeout on Gunicorn and proxy_connect_timeout, proxy_read_timeout on Nginx by writing a large number (120000) and now it’s better, I get timeout after about 90 seconds instead of 30 (default for Gunicorn) but still is not what I expected and it’s not enough to finish my job.
Also I don’t like so much this approach. I don’t want infinite timeout for every request.
What’s the best approach to deal with long request and not have timeout?
Maybe by answering to user with a message and then run the job in background? Any suggestion?
Thanks.

Asked By: Alessio

||

Source

Answer 1

Using Celery With Django for Background Task Processing that means process the CSV file with celery async task.

OR

As a quick hack, if you don’t want to use celery; use multi-threading to process the CSV and save the result of the processing in DB or file and server the result from DB or file.

Note: Never Process big files on main thread; always try to use a different server to process the big files.If different server is not possible then try to process it in background task

Many solutions can be found on this StackOverflow link

Answered By: Himanshu dua

Answer 2

    I spend more time to increase the request timeout. And Finally I got the soluton
    For **Django Rest API ( Nginx + supervisor + Gunicorn + AWS )**

     1. Add timeout in the gunicorn(Supervisor config)

        [program:gunicorn]
         command = <env_path>/env/bin/gunicorn --pythonpath=<python_path> --name=<nginx_name> --bind unix:/tmp/myproject.sock --workers 3 --graceful-timeout=900 --timeout=900 <project_path>.wsgi:application
        directory=<project_path>
        autostart=true
        autorestart=true
        stopsignal=TERM
        stdout_logfile=/var/log/gunicorn.stdout.log
        stdout_logfile_maxbytes=1MB
        stdout_logfile_backups=5
        stderr_logfile=/var/log/gunicorn.stderr.log
        stderr_logfile_maxbytes=1MB
        stderr_logfile_backups=5

     2. Add proxy on nginx file

     proxy_connect_timeout       900;
     proxy_send_timeout          900;
     proxy_read_timeout          900;
     send_timeout                900;

     3. Changes in Load Balancer ( If you configure  -> in EC2 instance)
 [Load Balancer][1]


  [1]: https://i.stack.imgur.com/uSUrK.png

Answered By: Rajan Mandanka

Django long request timeout

Question:

Answers: