Fetch a file from a local url with Python requests?

packages/urllib3/poolmanager.py pretty much explains it. Requests doesn’t support local url.

pool_classes_by_scheme = {                                                        
    'http': HTTPConnectionPool,                                                   
    'https': HTTPSConnectionPool,                                              
}                                                                                 

In a recent project, I’ve had the same issue. Since requests doesn’t support the “file” scheme, I’ll patch our code to load the content locally. First, I define a function to replace requests.get:

def local_get(self, url):
    "Fetch a stream from local files."
    p_url = six.moves.urllib.parse.urlparse(url)
    if p_url.scheme != 'file':
        raise ValueError("Expected file scheme")

    filename = six.moves.urllib.request.url2pathname(p_url.path)
    return open(filename, 'rb')

Then, somewhere in test setup or decorating the test function, I use mock.patch to patch the get function on requests:

@mock.patch('requests.get', local_get)
def test_handle_remote_file(self):
    ...

This technique is somewhat brittle — it doesn’t help if the underlying code calls requests.request or constructs a Session and calls that. There may be a way to patch requests at a lower level to support file: URLs, but in my initial investigation, there didn’t seem to be an obvious hook point, so I went with this simpler approach.

Here’s a transport adapter I wrote which is more featureful than b1r3k’s and has no additional dependencies beyond Requests itself. I haven’t tested it exhaustively yet, but what I have tried seems to be bug-free.

import requests
import os, sys

if sys.version_info.major < 3:
    from urllib import url2pathname
else:
    from urllib.request import url2pathname

class LocalFileAdapter(requests.adapters.BaseAdapter):
    """Protocol Adapter to allow Requests to GET file:// URLs

    @todo: Properly handle non-empty hostname portions.
    """

    @staticmethod
    def _chkpath(method, path):
        """Return an HTTP status for the given filesystem path."""
        if method.lower() in ('put', 'delete'):
            return 501, "Not Implemented"  # TODO
        elif method.lower() not in ('get', 'head'):
            return 405, "Method Not Allowed"
        elif os.path.isdir(path):
            return 400, "Path Not A File"
        elif not os.path.isfile(path):
            return 404, "File Not Found"
        elif not os.access(path, os.R_OK):
            return 403, "Access Denied"
        else:
            return 200, "OK"

    def send(self, req, **kwargs):  # pylint: disable=unused-argument
        """Return the file specified by the given request

        @type req: C{PreparedRequest}
        @todo: Should I bother filling `response.headers` and processing
               If-Modified-Since and friends using `os.stat`?
        """
        path = os.path.normcase(os.path.normpath(url2pathname(req.path_url)))
        response = requests.Response()

        response.status_code, response.reason = self._chkpath(req.method, path)
        if response.status_code == 200 and req.method.lower() != 'head':
            try:
                response.raw = open(path, 'rb')
            except (OSError, IOError) as err:
                response.status_code = 500
                response.reason = str(err)

        if isinstance(req.url, bytes):
            response.url = req.url.decode('utf-8')
        else:
            response.url = req.url

        response.request = req
        response.connection = self

        return response

    def close(self):
        pass

(Despite the name, it was completely written before I thought to check Google, so it has nothing to do with b1r3k’s.) As with the other answer, follow this with:

requests_session = requests.session()
requests_session.mount('file://', LocalFileAdapter())
r = requests_session.get('file:///path/to/your/file')

The easiest way seems using requests-file.

https://github.com/dashea/requests-file (available through PyPI too)

"Requests-File is a transport adapter for use with the Requests Python library to allow local filesystem access via file:// URLs."

This in combination with requests-html is pure magic 🙂

To load a file from a local URL, e.g. an image file you can do this:

import urllib
from PIL import Image

Image.open(urllib.request.urlopen('file:///path/to/your/file.png'))

For anyone who wants the bare minimum you have to do to get this to work, and with no other external dependencies, this is it:

import requests
from requests.adapters import HTTPAdapter
from urllib3.response import HTTPResponse

class FileAdapter(HTTPAdapter):
    def send(self, request, *args, **kwargs):
        resp = HTTPResponse(body=open(request.url[7:], 'rb'), status=200, preload_content=False)
        return self.build_response(request, resp)

session = requests.Session()
session.mount('file://', FileAdapter())
resp = session.get('file:///path/to/file.ext')