Python requests SSL error – certificate verify failed
Question:
This code
import requests
requests.get("https://hcaidcs.phe.org.uk/WebPages/GeneralHomePage.aspx")
is giving me this error
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
I know practically nothing about SSL, but I’ve tried downloading the site’s certificate and pointing to that file using the verify
option, but it hasn’t worked. Am I missing something?
Answers:
import requests
html = requests.get("https://hcaidcs.phe.org.uk/WebPages/GeneralHomePage.aspx",verify=False).text
You should write it like this, and I’ve verified it
As already pointed out in a comment: the site has a bad SSL implementation as can be seen from the SSLLabs report. The main part of this report regarding your problem is:
This server’s certificate chain is incomplete. Grade capped to B.
This means that the server is not sending the full certificate chain as is needed to verify the certificate. This means you need to add the missing certificates yourself when validating. For this you need to include the PEM for the missing chain certificate C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert SHA2 High Assurance Server CA and also for the root CA C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert High Assurance EV Root CA info a file my_trust_store.pem
and then you can call:
requests.get("https://...", verify='my_trust_store.pem')
… but I’ve tried downloading the site’s certificate and pointing to that file using the verify option
This will not work with normal leaf certificates. Since the SSL stack of Python is based on OpenSSL and OpenSSL expects only trusted certificate authorities in the trust store (i.e. given with verify
) and a server certificate is not CA certificate it will not help to add it to the trust store.
cat institution-certificate.pem >> venv/lib/python3.9/site-packages/certifi/cacert.pem
This should solve the problem if your network requires a CA
If you can avoid the certificate verification (not secure), set PYTHONHTTPSVERIFY environment variable to 0:
export PYTHONHTTPSVERIFY=0
This will skip the certificate verification.
using the certifi doesn’t seem to be implied, so i’ll show you what made my solution:
import urllib, urllib2, ssl
import certifi
request = urllib2.Request(url=url)
kw = dict()
if url.startswith('https://'):
certifi_context = ssl.create_default_context(cafile=certifi.where())
kw.update(context=certifi_context)
urllib2.urlopen(request, **kw)
This code
import requests
requests.get("https://hcaidcs.phe.org.uk/WebPages/GeneralHomePage.aspx")
is giving me this error
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)
I know practically nothing about SSL, but I’ve tried downloading the site’s certificate and pointing to that file using the verify
option, but it hasn’t worked. Am I missing something?
import requests
html = requests.get("https://hcaidcs.phe.org.uk/WebPages/GeneralHomePage.aspx",verify=False).text
You should write it like this, and I’ve verified it
As already pointed out in a comment: the site has a bad SSL implementation as can be seen from the SSLLabs report. The main part of this report regarding your problem is:
This server’s certificate chain is incomplete. Grade capped to B.
This means that the server is not sending the full certificate chain as is needed to verify the certificate. This means you need to add the missing certificates yourself when validating. For this you need to include the PEM for the missing chain certificate C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert SHA2 High Assurance Server CA and also for the root CA C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert High Assurance EV Root CA info a file my_trust_store.pem
and then you can call:
requests.get("https://...", verify='my_trust_store.pem')
… but I’ve tried downloading the site’s certificate and pointing to that file using the verify option
This will not work with normal leaf certificates. Since the SSL stack of Python is based on OpenSSL and OpenSSL expects only trusted certificate authorities in the trust store (i.e. given with verify
) and a server certificate is not CA certificate it will not help to add it to the trust store.
cat institution-certificate.pem >> venv/lib/python3.9/site-packages/certifi/cacert.pem
This should solve the problem if your network requires a CA
If you can avoid the certificate verification (not secure), set PYTHONHTTPSVERIFY environment variable to 0:
export PYTHONHTTPSVERIFY=0
This will skip the certificate verification.
using the certifi doesn’t seem to be implied, so i’ll show you what made my solution:
import urllib, urllib2, ssl
import certifi
request = urllib2.Request(url=url)
kw = dict()
if url.startswith('https://'):
certifi_context = ssl.create_default_context(cafile=certifi.where())
kw.update(context=certifi_context)
urllib2.urlopen(request, **kw)