How to get Python requests to trust a self signed SSL certificate?
Question:
import requests
data = {'foo':'bar'}
url = 'https://foo.com/bar'
r = requests.post(url, data=data)
If the URL uses a self signed certificate, this fails with
requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
I know that I can pass False to the verify parameter, like this:
r = requests.post(url, data=data, verify=False)
However, what I would like to do is point requests to a copy of the public key on disk and tell it to trust that certificate.
Answers:
try:
r = requests.post(url, data=data, verify='/path/to/public_key.pem')
With the verify parameter you can provide a custom certificate authority bundle
requests.get(url, verify=path_to_bundle_file)
From the docs:
You can pass verify the path to a CA_BUNDLE file with certificates of
trusted CAs. This list of trusted CAs can also be specified through
the REQUESTS_CA_BUNDLE environment variable.
The easiest is to export the variable REQUESTS_CA_BUNDLE that points to your private certificate authority, or a specific certificate bundle. On the command line you can do that as follows:
export REQUESTS_CA_BUNDLE=/path/to/your/certificate.pem
python script.py
If you have your certificate authority and you don’t want to type the export each time you can add the REQUESTS_CA_BUNDLE to your ~/.bash_profile as follows:
echo "export REQUESTS_CA_BUNDLE=/path/to/your/certificate.pem" >> ~/.bash_profile ; source ~/.bash_profile
Case where multiple certificates are needed was solved as follows:
Concatenate the multiple root pem files, myCert-A-Root.pem and myCert-B-Root.pem, to a file. Then set the requests REQUESTS_CA_BUNDLE var to that file in my ./.bash_profile.
$ cp myCert-A-Root.pem ca_roots.pem
$ cat myCert-B-Root.pem >> ca_roots.pem
$ echo "export REQUESTS_CA_BUNDLE=~/PATH_TO/CA_CHAIN/ca_roots.pem" >> ~/.bash_profile ; source ~/.bash_profile
Incase anyone happens to land here (like I did) looking to add a CA (in my case Charles Proxy) for httplib2, it looks like you can append it to the cacerts.txt file included with the python package.
For example:
cat ~/Desktop/charles-ssl-proxying-certificate.pem >> /usr/local/google-cloud-sdk/lib/third_party/httplib2/cacerts.txt
The environment variables referenced in other solutions appear to be requests-specific and were not picked up by httplib2 in my testing.
Setting export SSL_CERT_FILE=/path/file.crt should do the job.
You may try:
settings = s.merge_environment_settings(prepped.url, None, None, None, None)
You can read more here: http://docs.python-requests.org/en/master/user/advanced/
If you’re behind a corporate network firewall like I was, ask your network admin where your corporate certificates are, then:
import os
os.environ["REQUESTS_CA_BUNDLE"] = 'path/to/corporate/cert.pem'
os.environ["SSL_CERT_FILE"] = 'path/to/corporate/cert.pem'
This fixed issues I had with requests and openssl.
All of the answers to this question point to the same path: get the PEM file, but they don’t tell you how to get it from the website itself.
Getting the PEM file from the website itself is a valid option if you trust the site, such as on an internal corporate server. If you trust the site, why should you do this? You should do this because it helps protect yourself and others from inadvertently re-using your code on a site that isn’t safe.
Here is how you can get the PEM file.
-
Click on the lock next to the url.
-
Navigate to where you can see the certificates and open the certificates.
-
Download the PEM CERT chain.
-
Put the .PEM file somewhere you script can access it and try verify=r"pathtopem_chain.pem" within your requests call.
r = requests.get(url, verify='pathtopublic_key.pem')
In a dev environment, using Poetry as virtual env provider on a Mac with Python 3.8 I used this answer https://stackoverflow.com/a/42982144/15484549 as base and appended the content of my self-signed root certificate to the certifi cacert.pem file.
The steps in detail:
cd project_folder
poetry add requests
# or if you use something else, make sure certifi is among the dependencies
poetry shell
python
>>> import certifi
>>> certifi.where()
/path/to/the/certifi/cacert.pem
>>> exit()
cat /path/to/self-signed-root-cert.pem >> /path/to/the/certifi/cacert.pem
python the_script_you_want_to_run.py
I know it is an old thread. However, I run into this issue recently. My python requests code does not accept the self-signed certificate but curl does. It turns out python requests are very strict on the self-signed certificate. It needs to be a root CA certificate. In other words,
Basic Constraints: CA:TRUE
Key Usage: Digital Signature, Non Repudiation, Key Encipherment, Certificate Sign
import requests
data = {'foo':'bar'}
url = 'https://foo.com/bar'
r = requests.post(url, data=data)
If the URL uses a self signed certificate, this fails with
requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
I know that I can pass False to the verify parameter, like this:
r = requests.post(url, data=data, verify=False)
However, what I would like to do is point requests to a copy of the public key on disk and tell it to trust that certificate.
try:
r = requests.post(url, data=data, verify='/path/to/public_key.pem')
With the verify parameter you can provide a custom certificate authority bundle
requests.get(url, verify=path_to_bundle_file)
From the docs:
You can pass
verifythe path to a CA_BUNDLE file with certificates of
trusted CAs. This list of trusted CAs can also be specified through
the REQUESTS_CA_BUNDLE environment variable.
The easiest is to export the variable REQUESTS_CA_BUNDLE that points to your private certificate authority, or a specific certificate bundle. On the command line you can do that as follows:
export REQUESTS_CA_BUNDLE=/path/to/your/certificate.pem
python script.py
If you have your certificate authority and you don’t want to type the export each time you can add the REQUESTS_CA_BUNDLE to your ~/.bash_profile as follows:
echo "export REQUESTS_CA_BUNDLE=/path/to/your/certificate.pem" >> ~/.bash_profile ; source ~/.bash_profile
Case where multiple certificates are needed was solved as follows:
Concatenate the multiple root pem files, myCert-A-Root.pem and myCert-B-Root.pem, to a file. Then set the requests REQUESTS_CA_BUNDLE var to that file in my ./.bash_profile.
$ cp myCert-A-Root.pem ca_roots.pem
$ cat myCert-B-Root.pem >> ca_roots.pem
$ echo "export REQUESTS_CA_BUNDLE=~/PATH_TO/CA_CHAIN/ca_roots.pem" >> ~/.bash_profile ; source ~/.bash_profile
Incase anyone happens to land here (like I did) looking to add a CA (in my case Charles Proxy) for httplib2, it looks like you can append it to the cacerts.txt file included with the python package.
For example:
cat ~/Desktop/charles-ssl-proxying-certificate.pem >> /usr/local/google-cloud-sdk/lib/third_party/httplib2/cacerts.txt
The environment variables referenced in other solutions appear to be requests-specific and were not picked up by httplib2 in my testing.
Setting export SSL_CERT_FILE=/path/file.crt should do the job.
You may try:
settings = s.merge_environment_settings(prepped.url, None, None, None, None)
You can read more here: http://docs.python-requests.org/en/master/user/advanced/
If you’re behind a corporate network firewall like I was, ask your network admin where your corporate certificates are, then:
import os
os.environ["REQUESTS_CA_BUNDLE"] = 'path/to/corporate/cert.pem'
os.environ["SSL_CERT_FILE"] = 'path/to/corporate/cert.pem'
This fixed issues I had with requests and openssl.
All of the answers to this question point to the same path: get the PEM file, but they don’t tell you how to get it from the website itself.
Getting the PEM file from the website itself is a valid option if you trust the site, such as on an internal corporate server. If you trust the site, why should you do this? You should do this because it helps protect yourself and others from inadvertently re-using your code on a site that isn’t safe.
Here is how you can get the PEM file.
-
Click on the lock next to the url.
-
Navigate to where you can see the certificates and open the certificates.
-
Download the PEM CERT chain.
-
Put the .PEM file somewhere you script can access it and try
verify=r"pathtopem_chain.pem"within yourrequestscall.
r = requests.get(url, verify='pathtopublic_key.pem')
In a dev environment, using Poetry as virtual env provider on a Mac with Python 3.8 I used this answer https://stackoverflow.com/a/42982144/15484549 as base and appended the content of my self-signed root certificate to the certifi cacert.pem file.
The steps in detail:
cd project_folder
poetry add requests
# or if you use something else, make sure certifi is among the dependencies
poetry shell
python
>>> import certifi
>>> certifi.where()
/path/to/the/certifi/cacert.pem
>>> exit()
cat /path/to/self-signed-root-cert.pem >> /path/to/the/certifi/cacert.pem
python the_script_you_want_to_run.py
I know it is an old thread. However, I run into this issue recently. My python requests code does not accept the self-signed certificate but curl does. It turns out python requests are very strict on the self-signed certificate. It needs to be a root CA certificate. In other words,
Basic Constraints: CA:TRUE
Key Usage: Digital Signature, Non Repudiation, Key Encipherment, Certificate Sign