How to replace hyperlinks in StreamingResponse?

Question

Is that possible to replace hyperlinks in StreamingResponse?

I’m using below code to stream HTML content.

from starlette.requests import Request
from starlette.responses import StreamingResponse
from starlette.background import BackgroundTask

import httpx

client = httpx.AsyncClient(base_url="http://containername:7800/")


async def _reverse_proxy(request: Request):
    url = httpx.URL(path=request.url.path, query=request.url.query.encode("utf-8"))
    rp_req = client.build_request(
        request.method, url, headers=request.headers.raw, content=await request.body()
    )
    rp_resp = await client.send(rp_req, stream=True)
    return StreamingResponse(
        rp_resp.aiter_raw(),
        status_code=rp_resp.status_code,
        headers=rp_resp.headers,
        background=BackgroundTask(rp_resp.aclose),
    )


app.add_route("/titles/{path:path}", _reverse_proxy, ["GET", "POST"])

It’s working fine, but I would like to replace a href links.
Is that possible?

I’ve tired to wrap the generator like below:

async def adjust_response(iterable):
    # Adjust hyperlinks in response.
    async for element in iterable.aiter_raw():
        yield element.decode("utf-8").replace("/admin", "/gateway/admins/SERVICE_A").encode("utf-8")

but this caused that error:

h11._util.LocalProtocolError: Too much data for declared Content-Length

Asked By: Mateusz

||

Source

Answer 1

One solution would clearly be to read from the original response generator (as mentioned in the comments section above), modify each href link, and then yield the modified content.

Another solution would be to use JavaScript to find all links in the HTML document and modify them accordingly. If you had access to the external service’s HTML files, you could just add a script to modify all the href links, only if the Window.location is not pointing to the service’s host (e.g., if (window.location.host != "containername:7800" ) {...}). Even though you don’t have access to the external HTML files, you could still do that on server side. You can create a StaticFiles instance to serve a replace.js script file, and simply inject that script using a <script> tag in the <head> section of the HTML page (Note: if no <head> tag is provided, then find the <html> tag and create the <head></head> with the <script> in it). You can have the script run when the whole page has loaded, using window.onload event, or, preferably, when the initial HTML document has been completely loaded and parsed (without waiting for stylesheets, images, etc., to finish loading) using DOMContentLoaded event. Using this approach, you don’t have to go through each chunk to modify each href link on server side, but rather inject the script and then have the replacement taking place on client side.

On a side note, if the incoming request has a rather large body that couldn’t fit into RAM (for instance, if large files are included in the request) and would cause your application to slow down or even crash, then instead of reading the entire body into RAM using await request.body(), read it in chunks using Starlette’s stream() method (see this answer and this answer), which returns an async bytes generator (see httpx‘s Streaming requests documentation as well); hence, you could use: client.build_request(..., content=request.stream()).

Working Example:

# ...
from fastapi.staticfiles import StaticFiles

app = FastAPI()
app.mount("/static-js", StaticFiles(directory="static-js"), name="static-js")

client = httpx.AsyncClient(base_url="http://containername:7800/")

async def iter_content(r):
    found = False
    async for chunk in r.aiter_raw():
        if not found:
            idx = chunk.find(bytes('<head>', 'utf-8'))
            if idx != -1:
                found = True
                b_arr = bytearray(chunk)
                b_arr[idx+6:] = bytes('<script src="/static-js/replace.js"></script>', 'utf-8') + b_arr[idx+6:]
                chunk = bytes(b_arr)
        yield chunk

async def _reverse_proxy(request: Request):
    url = httpx.URL(path=request.url.path, query=request.url.query.encode("utf-8"))
    rp_req = client.build_request(
        request.method, url, headers=request.headers.raw, content=await request.body()
    )
    rp_resp = await client.send(rp_req, stream=True)
    return StreamingResponse(
        iter_content(rp_resp),
        status_code=rp_resp.status_code,
        headers=rp_resp.headers,
        background=BackgroundTask(rp_resp.aclose),
    )

app.add_route("/titles/{path:path}", _reverse_proxy, ["GET", "POST"])

The JS script (replace.js):

document.addEventListener('DOMContentLoaded', (event) => {
   var anchors = document.getElementsByTagName("a");

   for (var i = 0; i < anchors.length; i++) {
      let path = anchors[i].pathname.replace('/admin', '/admins/SERVICE_A');
      anchors[i].href = path + anchors[i].search + anchors[i].hash;
   }
});

Answered By: Chris

How to replace hyperlinks in StreamingResponse?

Question:

Answers:

Working Example: