How do you add environment variables to a python scrapy project? dotenv didn't work

Question:

I’m having trouble incorporating an IP address into a format string in my Python Scrapy project. I was trying to use python-dotenv to store sensitive information, such as server IPs, in a .env file and load it into my project, instead of hardcoding it.

I added python-dotenv to the settings.py file of my Scrapy project, but when I run a function that should use the values stored in os, I get an error saying that it can’t detect dotenv. Can someone help me understand why this is happening and how to properly incorporate an IP address in a format string using python-dotenv in a Python Scrapy project?

Asked By: rom

||
Source

Answers:

It is difficult to help figure out what you might be doing wrong with the limited information you have provided. However here is a general example of how one might use python-dotenv with a scrapy settings.py file.

First create a .env file and add your IP

.env

IPADDRESS = SUPERSECRETIPADDRESS

Finally in your settings.py file you need to import dotenv and then run dotenv.load_dotenv() and then you can get the ip address from the environment variable.

settings.py

import os
import dotenv

dotenv.load_dotenv()

IP_ADDRESS_SCRAPY_SETTING = os.environ["IPADDRESS"]

print(IP_ADDRESS_SCRAPY_SETTING)

output:

SUPERSECRETIPADDRESS

Note: Make sure that the .env file is in the same directory as the settings.py file or it is in one of it’s parent/ancestor directories.

So if your settings.py file is in

  • /home/username/scrapy_project/scrapy_project/settings.py

then that means the .env file can be in one of the following:

  • /home/username/scrapy_project/scrapy_project/.env
  • /home/username/scrapy_project/.env
  • /home/username/.env
  • /home/.env
  • /.env

Otherwise it will not be able to find the file.

Answered By: Alexander