Dryscrape "Can Not Find XVFB" Error on Nginx & Ubuntu Server 18.04

Question:

I need a help about Dryscrape and XVFB. I’m using Dryscrape on my Flask application for web scraping from a javascript page. If I serve it with python3 main.py and the app config app.run(host="0.0.0.0") everything works great at port 5000. But I have to use Nginx and uWSGI to serve my application on a domain. So, while I’m serving the app with Nginx, I get this error.

Sep 30 19:39:16 /address uwsgi[959]:   File "./scrape_manager.py", line 10, in get_details
Sep 30 19:39:16 /address uwsgi[959]:     dryscrape.start_xvfb()
Sep 30 19:39:16 /address uwsgi[959]:   File "/home/user/app/venv/lib/python3.6/site-packages/dryscrape/xvfb.py", line 8, in start_xvfb
Sep 30 19:39:16 /address uwsgi[959]:     xvfb = Xvfb()
Sep 30 19:39:16 /address uwsgi[959]:   File "/home/user/app/venv/lib/python3.6/site-packages/xvfbwrapper.py", line 41, in __init__
Sep 30 19:39:16 /address uwsgi[959]:     raise EnvironmentError(msg)

What should I do to fix this?

Asked By: rch

||
Source

Answers:

I found the solution.

The problem was in the service file of Nginx.

app.service

[Unit]
Description=uWSGI instance to serve app
After=network.target

[Service]
User=user
Group=www-data
WorkingDirectory=/home/user/app/app
Environment="PATH=/home/user/app/venv/bin"
ExecStart=/home/user/app/venv/bin/uwsgi --ini /home/user/app/app/app.ini

[Install]
WantedBy=multi-user.target

Actually the source of failure was the environment path, it’s only one and it’s the virtual environment of the application. But Dryscrape needs to access XVFB from /bin or /usr/bin (check it, where does it exist). In this way, I had to extend environment paths with both to try and I added /usr/bin and /bin

[Unit]
Description=uWSGI instance to serve app
After=network.target

[Service]
User=user
Group=www-data
WorkingDirectory=/home/user/app/app
Environment="PATH=/home/user/app/venv/bin:/usr/bin:/bin"
ExecStart=/home/user/app/venv/bin/uwsgi --ini /home/user/app/app/app.ini

[Install]
WantedBy=multi-user.target

This is the correct service file but one of them (/bin or /usr/bin) is may be unnecessary so you have to check the paths, where XVFB file executing from. If not sure, add /usr/bin and /bin too; it’ll work but if you are able to find the path, add only one like this below.

Environment="PATH=/home/user/app/venv/bin:/usr/bin"

And also this status might be a tricky related to your user-group / user permissions too.

Answered By: rch