Apache SetEnv not working as expected with mod_wsgi
Question:
In a flask application I wrote, I make use of an external library which can be configured using environment variables. Note: I wrote this external library myself. So I could make changes if necessary. When running from the command line an running the flask server with:
# env = python virtual environment
ENV_VAR=foo ./env/bin/python myapp/webui.py
it all woks as expected. But after deploying it to apache, and using SetEnv it does not work anymore. In fact, printing out os.environ to stderr (so it shows up in the apache logs reveals, that the wsgi process seems to be in a very different environment (for one, os.environ['PWD'] seems to be way off. In fact, it points to my development folder.
To help identifying the problem, following are the relevant parts of the application as a standalone hello-world app. The error output, and observations are at the very end of the post.
App folder layout:
Python app:
.
├── myapp.ini
├── setup.py
└── testenv
├── __init__.py
├── model
│ └── __init__.py
└── webui.py
Apache folder (/var/www/michel/testenv):
.
├── env
│ ├── [...]
├── logs
│ ├── access.log
│ └── error.log
└── wsgi
└── app.wsgi
myapp.ini
[app]
somevar=somevalue
setup.py
from setuptools import setup, find_packages
setup(
name="testenv",
version='1.0dev1',
description="A test app",
long_description="Hello World!",
author="Some Author",
author_email="[email protected]",
license="BSD",
include_package_data=True,
install_requires = [
'flask',
],
packages=find_packages(exclude=["tests.*", "tests"]),
zip_safe=False,
)
testenv/init.py
# empty
testenv/model/init.py
from os.path import expanduser, join, exists
from os import getcwd, getenv, pathsep
import logging
import sys
__version__ = '1.0dev1'
LOG = logging.getLogger(__name__)
def find_config():
"""
Searches for an appropriate config file. If found, return the filename, and
the parsed search path
"""
path = [getcwd(), expanduser('~/.mycompany/myapp'), '/etc/mycompany/myapp']
env_path = getenv("MYAPP_PATH")
config_filename = getenv("MYAPP_CONFIG", "myapp.ini")
if env_path:
path = env_path.split(pathsep)
detected_conf = None
for dir in path:
conf_name = join(dir, config_filename)
if exists(conf_name):
detected_conf = conf_name
break
return detected_conf, path
def load_config():
"""
Load the config file.
Raises an OSError if no file was found.
"""
from ConfigParser import SafeConfigParser
conf, path = find_config()
if not conf:
raise OSError("No config file found! Search path was %r" % path)
parser = SafeConfigParser()
parser.read(conf)
LOG.info("Loaded settings from %r" % conf)
return parser
try:
CONF = load_config()
except OSError, ex:
# Give a helpful message instead of a scary stack-trace
print >>sys.stderr, str(ex)
sys.exit(1)
testenv/webui.py
from testenv.model import CONF
from flask import Flask
app = Flask(__name__)
@app.route('/')
def index():
return "Hello World %s!" % CONF.get('app', 'somevar')
if __name__ == '__main__':
app.debue = True
app.run()
Apache config
<VirtualHost *:80>
ServerName testenv-test.my.fq.dn
ServerAlias testenv-test
WSGIDaemonProcess testenv user=michel threads=5
WSGIScriptAlias / /var/www/michel/testenv/wsgi/app.wsgi
SetEnv MYAPP_PATH /var/www/michel/testenv/config
<Directory /var/www/michel/testenv/wsgi>
WSGIProcessGroup testenv
WSGIApplicationGroup %{GLOBAL}
Order deny,allow
Allow from all
</Directory>
ErrorLog /var/www/michel/testenv/logs/error.log
LogLevel warn
CustomLog /var/www/michel/testenv/logs/access.log combined
</VirtualHost>
app.wsgi
activate_this = '/var/www/michel/testenv/env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from os import getcwd
import logging, sys
from testenv.webui import app as application
# You may want to change this if you are using another logging setup
logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
LOG = logging.getLogger(__name__)
LOG.debug('Current path: {0}'.format(getcwd()))
# Application config
application.debug = False
# vim: set ft=python :
Error and observations
This is the output of the apache error log.
[Thu Jan 26 10:48:15 2012] [error] No config file found! Search path was ['/home/users/michel', '/home/users/michel/.mycompany/myapp', '/etc/mycompany/myapp']
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] mod_wsgi (pid=17946): Target WSGI script '/var/www/michel/testenv/wsgi/app.wsgi' cannot be loaded as Python module.
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] mod_wsgi (pid=17946): SystemExit exception raised by WSGI script '/var/www/michel/testenv/wsgi/app.wsgi' ignored.
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] Traceback (most recent call last):
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] File "/var/www/michel/testenv/wsgi/app.wsgi", line 10, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] from testenv.webui import app as application
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] File "/var/www/michel/testenv/env/lib/python2.6/site-packages/testenv-1.0dev1-py2.6.egg/testenv/webui.py", line 1, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] from testenv.model import CONF
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] File "/var/www/michel/testenv/env/lib/python2.6/site-packages/testenv-1.0dev1-py2.6.egg/testenv/model/__init__.py", line 51, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] sys.exit(1)
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] SystemExit: 1
My first observation is that the environment variable MYAPP_PATH does not appear in os.environ (this is not visible in this output, but I tested it, and it’s not there!). As such, the config “resolver” falls back to the default path.
And my second observation is the search path for the config file lists /home/users/michel as return value of os.getcwd(). I was actually expecting something inside /var/www/michel/testenv.
My instinct tells me, that the way I am doing config resolution is not right. Mainly because code is executed at import-time. This leads me to the idea, that maybe the config-resolution code is executed before the WSGI environment is properly set up. Am I onto something there?
Short discussion / tangential question
How would you do the config resolution in this case? Given that the “model” sub-folder is in reality an external module, which should also work in non-wsgi applications, and should provide a method to configure a database connection.
Personally, I like the way I search for config files while still being able to override it. Only, the fact that code is executed at import-time is making my spider-senses tingling like crazy. The rationale behind this: Configuration handling is completely hidden (abstraction-barrier) by fellow developers using this module, and it “just works”. They just need to import the module (with an existing config-file of course) and can jump right in without knowing any DB details. This also gives them an easy way to work with different databases (dev/test/deployment) and switch between them easily.
Now, inside mod_wsgi it does not anymore 🙁
Update:
Just now, to test my above idea, I changed the webui.py to the following:
import os
from flask import Flask, jsonify
app = Flask(__name__)
@app.route('/')
def index():
return jsonify(os.environ)
if __name__ == '__main__':
app.debue = True
app.run()
The output on the webpage is the following:
{
LANG: "C",
APACHE_RUN_USER: "www-data",
APACHE_PID_FILE: "/var/run/apache2.pid",
PWD: "/home/users/michel/tmp/testenv",
APACHE_RUN_GROUP: "www-data",
PATH: "/usr/local/bin:/usr/bin:/bin",
HOME: "/home/users/michel/"
}
This shows the same environment as the one seen by other debugging methods. So my initial though was wrong. But now I realised something stranger yet. os.environment['PWD'] is set to the folder where I have my development files. This is not at all where the application is running. Stranger yet, os.getcwd() returns /home/users/michel? This is inconsistent with what I see in os.environ. Should it not be the same as os.environ['PWD']?
The most important problem remains though: Why is the value set by apache’s SetEnv (MYAPP_PATH in this case) not found in os.environ?
Answers:
Note that the WSGI environment is passed upon each request to the application in the environ argument of the application object. This environment is totally unrelated to the process environment which is kept in os.environ. The SetEnv directive has no effect on os.environ and there is no way through Apache configuration directives to affect what is in the process environment.
So you have to do something else other than getenviron or os.environ['PWD'] to get the MY_PATH from apache.
Flask adds the wsgi environ to the request, not app.environ, it is done by the underlaying werkzeug. So on each request to the application, apache will add the MYAPP_CONF key and youll access it wherever you can access request it seems as, request.environ.get('MYAPP_CONFIG') for example.
@rapadura answer is correct in that you don’t have direct access to the SetEnv values in your Apache config, but you can work around it.
If you add a wrapper around application in your app.wsgi file you can set the os.environ on each request. See the following modified app.wsgi for an example:
activate_this = '/var/www/michel/testenv/env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from os import environ, getcwd
import logging, sys
from testenv.webui import app as _application
# You may want to change this if you are using another logging setup
logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
LOG = logging.getLogger(__name__)
LOG.debug('Current path: {0}'.format(getcwd()))
# Application config
_application.debug = False
def application(req_environ, start_response):
environ['MYAPP_CONF'] = req_environ['MYAPP_CONF']
return _application(req_environ, start_response)
If you have set more environment variables in your Apache config, then you need to explicitly set each of them in the application wrapper function.
In a flask application I wrote, I make use of an external library which can be configured using environment variables. Note: I wrote this external library myself. So I could make changes if necessary. When running from the command line an running the flask server with:
# env = python virtual environment
ENV_VAR=foo ./env/bin/python myapp/webui.py
it all woks as expected. But after deploying it to apache, and using SetEnv it does not work anymore. In fact, printing out os.environ to stderr (so it shows up in the apache logs reveals, that the wsgi process seems to be in a very different environment (for one, os.environ['PWD'] seems to be way off. In fact, it points to my development folder.
To help identifying the problem, following are the relevant parts of the application as a standalone hello-world app. The error output, and observations are at the very end of the post.
App folder layout:
Python app:
.
├── myapp.ini
├── setup.py
└── testenv
├── __init__.py
├── model
│ └── __init__.py
└── webui.py
Apache folder (/var/www/michel/testenv):
.
├── env
│ ├── [...]
├── logs
│ ├── access.log
│ └── error.log
└── wsgi
└── app.wsgi
myapp.ini
[app]
somevar=somevalue
setup.py
from setuptools import setup, find_packages
setup(
name="testenv",
version='1.0dev1',
description="A test app",
long_description="Hello World!",
author="Some Author",
author_email="[email protected]",
license="BSD",
include_package_data=True,
install_requires = [
'flask',
],
packages=find_packages(exclude=["tests.*", "tests"]),
zip_safe=False,
)
testenv/init.py
# empty
testenv/model/init.py
from os.path import expanduser, join, exists
from os import getcwd, getenv, pathsep
import logging
import sys
__version__ = '1.0dev1'
LOG = logging.getLogger(__name__)
def find_config():
"""
Searches for an appropriate config file. If found, return the filename, and
the parsed search path
"""
path = [getcwd(), expanduser('~/.mycompany/myapp'), '/etc/mycompany/myapp']
env_path = getenv("MYAPP_PATH")
config_filename = getenv("MYAPP_CONFIG", "myapp.ini")
if env_path:
path = env_path.split(pathsep)
detected_conf = None
for dir in path:
conf_name = join(dir, config_filename)
if exists(conf_name):
detected_conf = conf_name
break
return detected_conf, path
def load_config():
"""
Load the config file.
Raises an OSError if no file was found.
"""
from ConfigParser import SafeConfigParser
conf, path = find_config()
if not conf:
raise OSError("No config file found! Search path was %r" % path)
parser = SafeConfigParser()
parser.read(conf)
LOG.info("Loaded settings from %r" % conf)
return parser
try:
CONF = load_config()
except OSError, ex:
# Give a helpful message instead of a scary stack-trace
print >>sys.stderr, str(ex)
sys.exit(1)
testenv/webui.py
from testenv.model import CONF
from flask import Flask
app = Flask(__name__)
@app.route('/')
def index():
return "Hello World %s!" % CONF.get('app', 'somevar')
if __name__ == '__main__':
app.debue = True
app.run()
Apache config
<VirtualHost *:80>
ServerName testenv-test.my.fq.dn
ServerAlias testenv-test
WSGIDaemonProcess testenv user=michel threads=5
WSGIScriptAlias / /var/www/michel/testenv/wsgi/app.wsgi
SetEnv MYAPP_PATH /var/www/michel/testenv/config
<Directory /var/www/michel/testenv/wsgi>
WSGIProcessGroup testenv
WSGIApplicationGroup %{GLOBAL}
Order deny,allow
Allow from all
</Directory>
ErrorLog /var/www/michel/testenv/logs/error.log
LogLevel warn
CustomLog /var/www/michel/testenv/logs/access.log combined
</VirtualHost>
app.wsgi
activate_this = '/var/www/michel/testenv/env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from os import getcwd
import logging, sys
from testenv.webui import app as application
# You may want to change this if you are using another logging setup
logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
LOG = logging.getLogger(__name__)
LOG.debug('Current path: {0}'.format(getcwd()))
# Application config
application.debug = False
# vim: set ft=python :
Error and observations
This is the output of the apache error log.
[Thu Jan 26 10:48:15 2012] [error] No config file found! Search path was ['/home/users/michel', '/home/users/michel/.mycompany/myapp', '/etc/mycompany/myapp']
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] mod_wsgi (pid=17946): Target WSGI script '/var/www/michel/testenv/wsgi/app.wsgi' cannot be loaded as Python module.
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] mod_wsgi (pid=17946): SystemExit exception raised by WSGI script '/var/www/michel/testenv/wsgi/app.wsgi' ignored.
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] Traceback (most recent call last):
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] File "/var/www/michel/testenv/wsgi/app.wsgi", line 10, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] from testenv.webui import app as application
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] File "/var/www/michel/testenv/env/lib/python2.6/site-packages/testenv-1.0dev1-py2.6.egg/testenv/webui.py", line 1, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] from testenv.model import CONF
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] File "/var/www/michel/testenv/env/lib/python2.6/site-packages/testenv-1.0dev1-py2.6.egg/testenv/model/__init__.py", line 51, in <module>
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] sys.exit(1)
[Thu Jan 26 10:48:15 2012] [error] [client 10.115.192.101] SystemExit: 1
My first observation is that the environment variable MYAPP_PATH does not appear in os.environ (this is not visible in this output, but I tested it, and it’s not there!). As such, the config “resolver” falls back to the default path.
And my second observation is the search path for the config file lists /home/users/michel as return value of os.getcwd(). I was actually expecting something inside /var/www/michel/testenv.
My instinct tells me, that the way I am doing config resolution is not right. Mainly because code is executed at import-time. This leads me to the idea, that maybe the config-resolution code is executed before the WSGI environment is properly set up. Am I onto something there?
Short discussion / tangential question
How would you do the config resolution in this case? Given that the “model” sub-folder is in reality an external module, which should also work in non-wsgi applications, and should provide a method to configure a database connection.
Personally, I like the way I search for config files while still being able to override it. Only, the fact that code is executed at import-time is making my spider-senses tingling like crazy. The rationale behind this: Configuration handling is completely hidden (abstraction-barrier) by fellow developers using this module, and it “just works”. They just need to import the module (with an existing config-file of course) and can jump right in without knowing any DB details. This also gives them an easy way to work with different databases (dev/test/deployment) and switch between them easily.
Now, inside mod_wsgi it does not anymore 🙁
Update:
Just now, to test my above idea, I changed the webui.py to the following:
import os
from flask import Flask, jsonify
app = Flask(__name__)
@app.route('/')
def index():
return jsonify(os.environ)
if __name__ == '__main__':
app.debue = True
app.run()
The output on the webpage is the following:
{
LANG: "C",
APACHE_RUN_USER: "www-data",
APACHE_PID_FILE: "/var/run/apache2.pid",
PWD: "/home/users/michel/tmp/testenv",
APACHE_RUN_GROUP: "www-data",
PATH: "/usr/local/bin:/usr/bin:/bin",
HOME: "/home/users/michel/"
}
This shows the same environment as the one seen by other debugging methods. So my initial though was wrong. But now I realised something stranger yet. os.environment['PWD'] is set to the folder where I have my development files. This is not at all where the application is running. Stranger yet, os.getcwd() returns /home/users/michel? This is inconsistent with what I see in os.environ. Should it not be the same as os.environ['PWD']?
The most important problem remains though: Why is the value set by apache’s SetEnv (MYAPP_PATH in this case) not found in os.environ?
Note that the WSGI environment is passed upon each request to the application in the environ argument of the application object. This environment is totally unrelated to the process environment which is kept in os.environ. The SetEnv directive has no effect on os.environ and there is no way through Apache configuration directives to affect what is in the process environment.
So you have to do something else other than getenviron or os.environ['PWD'] to get the MY_PATH from apache.
Flask adds the wsgi environ to the request, not app.environ, it is done by the underlaying werkzeug. So on each request to the application, apache will add the MYAPP_CONF key and youll access it wherever you can access request it seems as, request.environ.get('MYAPP_CONFIG') for example.
@rapadura answer is correct in that you don’t have direct access to the SetEnv values in your Apache config, but you can work around it.
If you add a wrapper around application in your app.wsgi file you can set the os.environ on each request. See the following modified app.wsgi for an example:
activate_this = '/var/www/michel/testenv/env/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from os import environ, getcwd
import logging, sys
from testenv.webui import app as _application
# You may want to change this if you are using another logging setup
logging.basicConfig(stream=sys.stderr, level=logging.DEBUG)
LOG = logging.getLogger(__name__)
LOG.debug('Current path: {0}'.format(getcwd()))
# Application config
_application.debug = False
def application(req_environ, start_response):
environ['MYAPP_CONF'] = req_environ['MYAPP_CONF']
return _application(req_environ, start_response)
If you have set more environment variables in your Apache config, then you need to explicitly set each of them in the application wrapper function.