Unit testing a python app that uses the requests library

You could use a mocking library such as Mocker to intercept the calls to the requests library and return specified results.

As a very simple example, consider this class which uses the requests library:

class MyReq(object):
    def doSomething(self):
        r = requests.get('https://api.github.com', auth=('user', 'pass'))
        return r.headers['content-type']

Here’s a unit test that intercepts the call to requests.get and returns a specified result for testing:

import unittest
import requests
import myreq

from mocker import Mocker, MockerTestCase

class MyReqTests(MockerTestCase):
    def testSomething(self):
        # Create a mock result for the requests.get call
        result = self.mocker.mock()
        result.headers
        self.mocker.result({'content-type': 'mytest/pass'})

        # Use mocker to intercept the call to requests.get
        myget = self.mocker.replace("requests.get")
        myget('https://api.github.com', auth=('user', 'pass'))
        self.mocker.result(result)

        self.mocker.replay()

        # Now execute my code
        r = myreq.MyReq()
        v = r.doSomething()

        # and verify the results
        self.assertEqual(v, 'mytest/pass')
        self.mocker.verify()

if __name__ == '__main__':
    unittest.main()

When I run this unit test I get the following result:

.
----------------------------------------------------------------------
Ran 1 test in 0.004s

OK

using mocker like in srgerg’s answer:

def replacer(method, endpoint, json_string):
    from mocker import Mocker, ANY, CONTAINS
    mocker = Mocker()
    result = mocker.mock()
    result.json()
    mocker.count(1, None)
    mocker.result(json_string)
    replacement = mocker.replace("requests." + method)
    replacement(CONTAINS(endpoint), params=ANY)
    self.mocker.result(result)
    self.mocker.replay()

For the requests library, this would intercept the request by method and endpoint you’re hitting and replace the .json() on the response with the json_string passed in.

If you use specifically requests try httmock. It’s wonderfully simple and elegant:

from httmock import urlmatch, HTTMock
import requests

# define matcher:
@urlmatch(netloc=r'(.*.)?google.com$')
def google_mock(url, request):
    return 'Feeling lucky, punk?'

# open context to patch
with HTTMock(google_mock):
    # call requests
    r = requests.get('http://google.com/')
print r.content  # 'Feeling lucky, punk?'

If you want something more generic (e.g. to mock any library making http calls) go for httpretty.

Almost as elegant:

import requests
import httpretty

@httpretty.activate
def test_one():
    # define your patch:
    httpretty.register_uri(httpretty.GET, "http://yipit.com/",
                        body="Find the best daily deals")
    # use!
    response = requests.get('http://yipit.com')
    assert response.text == "Find the best daily deals"

HTTPretty is far more feature-rich – it offers also mocking status code, streaming responses, rotating responses, dynamic responses (with a callback).

Missing from these answers is requests-mock.

From their page:

>>> import requests
>>> import requests_mock

As a context manager:

>>> with requests_mock.mock() as m:

...     m.get('http://test.com', text='data')
...     requests.get('http://test.com').text
...
'data'

Or as a decorator:

>>> @requests_mock.mock()
... def test_func(m):
...     m.get('http://test.com', text='data')
...     return requests.get('http://test.com').text
...
>>> test_func()
'data'

If you break out your response handler/parser into a separate function, you can work with requests.Response objects directly, without needing to mock the client-server interaction.

Code under test

from xml.dom import minidom
from requests.models import Response

def function_under_test(s3_response: Response):
    doc = minidom.parseString(s3_response.text)

    return (
        s3_response.status_code,
        doc.getElementsByTagName('Code').item(0).firstChild.data,
    )

Test code

import unittest
from io import BytesIO

class Test(unittest.TestCase):

    def test_it(self):
        s3_response = Response()
        s3_response.status_code = 404
        s3_response.raw = BytesIO(b"""<?xml version="1.0" encoding="UTF-8"?>
            <Error>
            <Code>NoSuchKey</Code>
            <Message>The resource you requested does not exist</Message>
            <Resource>/mybucket/myfoto.jpg</Resource> 
            <RequestId>4442587FB7D0A2F9</RequestId>
            </Error>
        """)

        parsed_response = function_under_test(s3_response)

        self.assertEqual(404, parsed_response[0])
        self.assertEqual("NoSuchKey", parsed_response[1])

There’s a library for this, if you want to write your test server with Flask: requests-flask-adaptor

You just have to be careful with the order of imports when monkeypatching.