Environment properties are not passed to application in Elastic Beanstalk
Question:
When deploying my Django project, database settings are not configured because 'RDS_HOSTNAME' in os.environ
returns false
. In fact no environment properties are available at the time of deployment. All these properties are available after the deployment.
Running /opt/elasticbeanstalk/bin/get-config environment
returns following:
{"DJANGO_SETTINGS_MODULE":"myApp.settings","PYTHONPATH":"/var/app/venv/staging-LQM1lest/bin:$PYTHONPATH","RDS_DB_NAME":"ebdb","RDS_HOSTNAME":"xxxx.amazonaws.com","RDS_PASSWORD":"xxxx","RDS_PORT":"xxxx","RDS_USERNAME":"xxxx"}
All RDS prefixed properties are set but still somehow os.environ
is unable to read it.
setting.py
file:
# [...]
if 'RDS_HOSTNAME' in os.environ:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': os.environ['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
# [...]
Do I have to make any changes to make these properties available at the time of deployment?
Answers:
Seems like this is a serious bug and AWS doesn’t care about it. There are few ways I came up with to make this work but all of them require logging into the EB environment and do some manual work.
Solution 1
As suggested in comment by hephalump
-
Create an AWS secret manager
-
Check IAM instance profile
in EB’s environment Configuration->Security->Edit
.
-
Then go to IAM user console and go to Roles
. From there you can attach policy to the instance profile for secret manager.
-
Once it’s done, deploy the project
-
Then login to the environment (eb ssh environment_name
).
-
Go to /var/app/current/
directory and run this command: source /var/app/venv/*/bin/activate
.
-
Finally run python3 manage.py migrate
.
Solution 2
-
Edit .bash_profile
and add export these variables at the end of the file:
export RDS_DB_NAME=your_dbname
export RDS_USERNAME=user
export RDS_PASSWORD=pass
export RDS_HOSTNAME=host_endpoint
export RDS_PORT=3306
-
Run source ~/.bash_profile
-
Now you can deploy your project.
Solution 3
- Set all environment properties in EB environment’s configuration. (Go to
Configuration->Software->Edit->Environment properties
and add the key and values).
2. Add this snippet at the beginning of settings.py
from pathlib import Path
import os
import subprocess
import ast
def get_environ_vars():
completed_process = subprocess.run(
['/opt/elasticbeanstalk/bin/get-config', 'environment'],
stdout=subprocess.PIPE,
text=True,
check=True
)
return ast.literal_eval(completed_process.stdout)
-
Go to Database section and replace it with this snippet
if 'RDS_HOSTNAME' in os.environ:
DATABASES = {
'default': {
' ENGINE': 'django.db.backends.mysql',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': os.environ['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
else:
env_vars = get_environ_vars()
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': env_vars['RDS_DB_NAME'],
'USER': env_vars['RDS_USERNAME'],
'PASSWORD': env_vars['RDS_PASSWORD'],
'HOST': env_vars['RDS_HOSTNAME'],
'PORT': env_vars['RDS_PORT'],
}
}
-
Deploy the project.
-
Login to the environment (eb ssh environment_name
).
-
Go to /var/app/current/
directory and run this command: source /var/app/venv/*/bin/activate
.
-
Finally run python3 manage.py migrate
.
Conclusion:
Solution 1 is little complex and secret manager is not free (30 days trial only).
Solution 2 is simplest one but I do not recommend tempering any file manually on EB.
Solution 3 is a clean solution which I will use. This solution also takes care of this bug fix in future.
To use Environment properties in the system eg. run Symfony commands (using envs) you can simply run this:
/opt/elasticbeanstalk/bin/get-config environment | jq -r "to_entries|map("export (.key)='(.value|tostring)'")|.[]" >> /home/ec2-user/.bash_profile
It will add all your props in .bash_profile so when you log in via SSH they will be all set. Of course it would be good to add this in .ebextensions/*.config under commands: section.
The official AWS solution for making environmental variables available from the EC2 level also fixes this problem:
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/
When deploying my Django project, database settings are not configured because 'RDS_HOSTNAME' in os.environ
returns false
. In fact no environment properties are available at the time of deployment. All these properties are available after the deployment.
Running /opt/elasticbeanstalk/bin/get-config environment
returns following:
{"DJANGO_SETTINGS_MODULE":"myApp.settings","PYTHONPATH":"/var/app/venv/staging-LQM1lest/bin:$PYTHONPATH","RDS_DB_NAME":"ebdb","RDS_HOSTNAME":"xxxx.amazonaws.com","RDS_PASSWORD":"xxxx","RDS_PORT":"xxxx","RDS_USERNAME":"xxxx"}
All RDS prefixed properties are set but still somehow os.environ
is unable to read it.
setting.py
file:
# [...]
if 'RDS_HOSTNAME' in os.environ:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': os.environ['RDS_DB_NAME'],
'USER': os.environ['RDS_USERNAME'],
'PASSWORD': os.environ['RDS_PASSWORD'],
'HOST': os.environ['RDS_HOSTNAME'],
'PORT': os.environ['RDS_PORT'],
}
}
# [...]
Do I have to make any changes to make these properties available at the time of deployment?
Seems like this is a serious bug and AWS doesn’t care about it. There are few ways I came up with to make this work but all of them require logging into the EB environment and do some manual work.
Solution 1
As suggested in comment by hephalump
-
Create an AWS secret manager
-
Check
IAM instance profile
in EB’s environmentConfiguration->Security->Edit
. -
Then go to IAM user console and go to
Roles
. From there you can attach policy to the instance profile for secret manager. -
Once it’s done, deploy the project
-
Then login to the environment (
eb ssh environment_name
). -
Go to
/var/app/current/
directory and run this command:source /var/app/venv/*/bin/activate
. -
Finally run
python3 manage.py migrate
.
Solution 2
-
Edit
.bash_profile
and add export these variables at the end of the file:export RDS_DB_NAME=your_dbname export RDS_USERNAME=user export RDS_PASSWORD=pass export RDS_HOSTNAME=host_endpoint export RDS_PORT=3306
-
Run
source ~/.bash_profile
-
Now you can deploy your project.
Solution 3
- Set all environment properties in EB environment’s configuration. (Go to
Configuration->Software->Edit->Environment properties
and add the key and values).
2. Add this snippet at the beginning of settings.py
from pathlib import Path
import os
import subprocess
import ast
def get_environ_vars():
completed_process = subprocess.run(
['/opt/elasticbeanstalk/bin/get-config', 'environment'],
stdout=subprocess.PIPE,
text=True,
check=True
)
return ast.literal_eval(completed_process.stdout)
-
Go to Database section and replace it with this snippet
if 'RDS_HOSTNAME' in os.environ: DATABASES = { 'default': { ' ENGINE': 'django.db.backends.mysql', 'NAME': os.environ['RDS_DB_NAME'], 'USER': os.environ['RDS_USERNAME'], 'PASSWORD': os.environ['RDS_PASSWORD'], 'HOST': os.environ['RDS_HOSTNAME'], 'PORT': os.environ['RDS_PORT'], } } else: env_vars = get_environ_vars() DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': env_vars['RDS_DB_NAME'], 'USER': env_vars['RDS_USERNAME'], 'PASSWORD': env_vars['RDS_PASSWORD'], 'HOST': env_vars['RDS_HOSTNAME'], 'PORT': env_vars['RDS_PORT'], } }
-
Deploy the project.
-
Login to the environment (
eb ssh environment_name
). -
Go to
/var/app/current/
directory and run this command:source /var/app/venv/*/bin/activate
. -
Finally run
python3 manage.py migrate
.
Conclusion:
Solution 1 is little complex and secret manager is not free (30 days trial only).
Solution 2 is simplest one but I do not recommend tempering any file manually on EB.
Solution 3 is a clean solution which I will use. This solution also takes care of this bug fix in future.
To use Environment properties in the system eg. run Symfony commands (using envs) you can simply run this:
/opt/elasticbeanstalk/bin/get-config environment | jq -r "to_entries|map("export (.key)='(.value|tostring)'")|.[]" >> /home/ec2-user/.bash_profile
It will add all your props in .bash_profile so when you log in via SSH they will be all set. Of course it would be good to add this in .ebextensions/*.config under commands: section.
The official AWS solution for making environmental variables available from the EC2 level also fixes this problem:
https://aws.amazon.com/premiumsupport/knowledge-center/elastic-beanstalk-env-variables-shell/