Python requests: POST request dropping Authorization header
Question:
I’m trying to make an API POST request using the Python requests library. I am passing through an Authorization header but when I try debugging, I can see that the header is being dropped. I have no idea what’s going on.
Here’s my code:
access_token = get_access_token()
bearer_token = base64.b64encode(bytes("'Bearer {}'".format(access_token)), 'utf-8')
headers = {'Content-Type': 'application/json', 'Authorization': bearer_token}
data = '{"FirstName" : "Jane", "LastName" : "Smith"}'
response = requests.post('https://myserver.com/endpoint', headers=headers, data=data)
As you can see above, I manually set the Authorization header in the request arguments, but it is missing the actual request’s headers:
{'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.4.3 CPython/2.7.9 Linux/4.1.19-v7+'}.
An additional piece of information is that if I change the POST request to a GET request, the Authorization header passes through normally!
Why would this library be dropping the header for POST requests and how do I get this to work?
Using v2.4.3 of the requests lib and Python 2.7.9
Answers:
This is what request documentation says:
Authorization headers set with headers= will be overridden if credentials are specified in .netrc, which in turn will be overridden by the auth= parameter.
Authorization headers will be removed if you get redirected off-host.
Are you getting redirected in your request?
If this is the case try disabling the redirect with this option in post request:
allow_redirects=False
you can try to use a custom authorization in headers.
Define a custom authentication class:
class MyAuth(requests.auth.AuthBase):
def __init__(self, bearer_token):
self.username = None
self.bearer_token = bearer_token
def __call__(self, r):
r.headers['Authorization'] = self.bearer_token
return r
then use this to send the request:
headers = {'Content-Type': 'application/json'}
data = '{"FirstName" : "Jane", "LastName" : "Smith"}'
response = requests.post('https://myserver.com/endpoint', headers=headers, auth=MyAuth(bearer_token), data=data)
If this works please accept the answer. Or if you still have issue, do let us know.
Hope this helps.
TLDR
The url you are requesting redirects POST requests to a different host, so the requests library drops the Authoriztion header in fear of leaking your credentials. To fix that you can override the responsible method in requests’ Session class.
Details
In requests 2.4.3, the only place where reqeuests removes the Authorization header is when a request is redirected to a different host. This is the relevant code:
if 'Authorization' in headers:
# If we get redirected to a new host, we should strip out any
# authentication headers.
original_parsed = urlparse(response.request.url)
redirect_parsed = urlparse(url)
if (original_parsed.hostname != redirect_parsed.hostname):
del headers['Authorization']
In newer versions of requests, the Authorization header will be dropped in additional cases (for example if the redirect is from a secure to a non-secure protocol).
So what probably happens in your case, is that your POST requests get redirected to a different host. The only way you can provide authentication for a redirected host using the requests library, is through a .netrc file. Sadly that will only allow you to use HTTP Basic Auth, which doesn’t help you much. In that case, the best solution is probably to subclass requests.Session and override this behavior, like so:
from requests import Session
class NoRebuildAuthSession(Session):
def rebuild_auth(self, prepared_request, response):
"""
No code here means requests will always preserve the Authorization
header when redirected.
Be careful not to leak your credentials to untrusted hosts!
"""
session = NoRebuildAuthSession()
response = session.post('https://myserver.com/endpoint', headers=headers, data=data)
Edit
I have opened a pull-request to the requests library on github to add a warning when this happens. It has been waiting for a second approval to be merged (three months already).
The first (and maybe the actual) problem I see is how you create bearer_token because you are not encoding only your token but also the authentication type 'Bearer'
As I understood you only need to encode the token and have to provide the blank authentication type + the encoded token within your request header:
bearer_token = str(base64.b64encode(access_token.encode()), "utf8")
headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer {}'.format(bearer_token)}
If it is (also) a redirection issue you could simply find out the correct location and make your request to this url or you might think about sending the access token within the body of your POST if the server is accepting this.
From the documentation: Requests will attempt to get the authentication credentials for the URL’s hostname from the user’s netrc file. The netrc file overrides raw HTTP authentication headers set with headers=.
If credentials for the hostname are found, the request is sent with HTTP Basic Auth.
If you are being redirected you could try using allow_redirects=false
To send the Authorization header in POST request using ‘request’ lib. in Python
simply use this :
requests.post('https://api.github.com/user', auth=('user', 'pass'))
It’s a basic auth.
Loop requests worked for me
response = do_request(url, access_tok, "GET", payload={}, headers={}, allow_redirect=False)
if response.status_code in range(300, 310):
new_response = do_request(response.headers['Location'], access_tok, "GET", payload={}, headers={},)
# print(new_response.status_code)
pprint(new_response.json())
My issue was having an entry in .netrc that overrides the Authorization header. Other answers mention .netrc but do not address it. The solution is to manually create a Session and set trust_env to False.
import requests
session = requests.Session()
session.trust_env = False
headers={'Authorization': f'Bearer {TOKEN}'}
session.post(url, headers=headers)
There is a GitHub issue to prevent this override.
I’m trying to make an API POST request using the Python requests library. I am passing through an Authorization header but when I try debugging, I can see that the header is being dropped. I have no idea what’s going on.
Here’s my code:
access_token = get_access_token()
bearer_token = base64.b64encode(bytes("'Bearer {}'".format(access_token)), 'utf-8')
headers = {'Content-Type': 'application/json', 'Authorization': bearer_token}
data = '{"FirstName" : "Jane", "LastName" : "Smith"}'
response = requests.post('https://myserver.com/endpoint', headers=headers, data=data)
As you can see above, I manually set the Authorization header in the request arguments, but it is missing the actual request’s headers:
{'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.4.3 CPython/2.7.9 Linux/4.1.19-v7+'}.
An additional piece of information is that if I change the POST request to a GET request, the Authorization header passes through normally!
Why would this library be dropping the header for POST requests and how do I get this to work?
Using v2.4.3 of the requests lib and Python 2.7.9
This is what request documentation says:
Authorization headers set with headers= will be overridden if credentials are specified in .netrc, which in turn will be overridden by the auth= parameter.
Authorization headers will be removed if you get redirected off-host.
Are you getting redirected in your request?
If this is the case try disabling the redirect with this option in post request:
allow_redirects=False
you can try to use a custom authorization in headers.
Define a custom authentication class:
class MyAuth(requests.auth.AuthBase):
def __init__(self, bearer_token):
self.username = None
self.bearer_token = bearer_token
def __call__(self, r):
r.headers['Authorization'] = self.bearer_token
return r
then use this to send the request:
headers = {'Content-Type': 'application/json'}
data = '{"FirstName" : "Jane", "LastName" : "Smith"}'
response = requests.post('https://myserver.com/endpoint', headers=headers, auth=MyAuth(bearer_token), data=data)
If this works please accept the answer. Or if you still have issue, do let us know.
Hope this helps.
TLDR
The url you are requesting redirects POST requests to a different host, so the requests library drops the Authoriztion header in fear of leaking your credentials. To fix that you can override the responsible method in requests’ Session class.
Details
In requests 2.4.3, the only place where reqeuests removes the Authorization header is when a request is redirected to a different host. This is the relevant code:
if 'Authorization' in headers: # If we get redirected to a new host, we should strip out any # authentication headers. original_parsed = urlparse(response.request.url) redirect_parsed = urlparse(url) if (original_parsed.hostname != redirect_parsed.hostname): del headers['Authorization']
In newer versions of requests, the Authorization header will be dropped in additional cases (for example if the redirect is from a secure to a non-secure protocol).
So what probably happens in your case, is that your POST requests get redirected to a different host. The only way you can provide authentication for a redirected host using the requests library, is through a .netrc file. Sadly that will only allow you to use HTTP Basic Auth, which doesn’t help you much. In that case, the best solution is probably to subclass requests.Session and override this behavior, like so:
from requests import Session
class NoRebuildAuthSession(Session):
def rebuild_auth(self, prepared_request, response):
"""
No code here means requests will always preserve the Authorization
header when redirected.
Be careful not to leak your credentials to untrusted hosts!
"""
session = NoRebuildAuthSession()
response = session.post('https://myserver.com/endpoint', headers=headers, data=data)
Edit
I have opened a pull-request to the requests library on github to add a warning when this happens. It has been waiting for a second approval to be merged (three months already).
The first (and maybe the actual) problem I see is how you create bearer_token because you are not encoding only your token but also the authentication type 'Bearer'
As I understood you only need to encode the token and have to provide the blank authentication type + the encoded token within your request header:
bearer_token = str(base64.b64encode(access_token.encode()), "utf8")
headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer {}'.format(bearer_token)}
If it is (also) a redirection issue you could simply find out the correct location and make your request to this url or you might think about sending the access token within the body of your POST if the server is accepting this.
From the documentation: Requests will attempt to get the authentication credentials for the URL’s hostname from the user’s netrc file. The netrc file overrides raw HTTP authentication headers set with headers=.
If credentials for the hostname are found, the request is sent with HTTP Basic Auth.
If you are being redirected you could try using allow_redirects=false
To send the Authorization header in POST request using ‘request’ lib. in Python
simply use this :
requests.post('https://api.github.com/user', auth=('user', 'pass'))
It’s a basic auth.
Loop requests worked for me
response = do_request(url, access_tok, "GET", payload={}, headers={}, allow_redirect=False)
if response.status_code in range(300, 310):
new_response = do_request(response.headers['Location'], access_tok, "GET", payload={}, headers={},)
# print(new_response.status_code)
pprint(new_response.json())
My issue was having an entry in .netrc that overrides the Authorization header. Other answers mention .netrc but do not address it. The solution is to manually create a Session and set trust_env to False.
import requests
session = requests.Session()
session.trust_env = False
headers={'Authorization': f'Bearer {TOKEN}'}
session.post(url, headers=headers)
There is a GitHub issue to prevent this override.