How to render Streamable image on React coming from FastAPI server?

Question:

I would like to render an image on React returned from FastAPI backend using StreamingResponse. The image is in the form of a numpy array, which is of cv2 type of object.

@app.post("/predict")
async def root(file: UploadFile = File(...)):
    global model
    global store_coordinates
    global store_faces
    global store_mesh

    content = await file.read()
    nparr = np.fromstring(content, np.uint8)
    bg_img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
    ......................
    for i in range(len(store_coordinates)):
            x, y, w, h = store_coordinates[i]
            bg_img [b:b + d, a:a + c] = store_mesh[i]
    
    res,im_png = cv2.imencode(".png", bg_img)
    return StreamingResponse(io.BytesIO(im_png.tobytes()), media_type="image/png")

Here, I have created an API endpoint in which the uploaded image is received using POST request, and a StreamableResponse(Image) is returned.
How can I render this returned response on React frontend?

React Code:

import React, { Component } from "react";
import axios from "axios";

class Detect extends Component {
  state = {
    title: "",
    content: "",
    image: null,
  };

  handleChange = (e) => {
    this.setState({
      [e.target.id]: e.target.value,
    });
  };

  handleImageChange = (e) => {
    this.setState({
      image: e.target.files[0],
    });
  };

  
  handleSubmit = (e) => {
    e.preventDefault();
    console.log(this.state);
    let form_data = new FormData();
    form_data.append("image", this.state.image, this.state.image.name);
    let url = "http://127.0.0.1:8000/predict";
    axios
      .post(url, form_data, {
        headers: {
          "content-type": "multipart/form-data",
        },
      })
      .then((res) => {
        console.log(res.data);
      })
      .catch((err) => console.log(err));
  };

  render() {
    return (
      <div className="App">
        <form onSubmit={this.handleSubmit}>
          <p>
            <input
              type="file"
              id="image"
              accept="image/png, image/jpeg"
              onChange={this.handleImageChange}
              required
            />
          </p>
          <input type="submit" />
        </form>
        <div id="image-render">
          <img></img>
        </div>
      </div>
    );
  }
}

export default Detect;

I would like to render the returned image in the div tag which has the id of image-render.

Edit – This is the response I get from my backend.

enter image description here

Asked By: Raj Shah

||
Source

Answers:

There two options:

  1. encode the image data to Base64 format on server side and
    return the base64-encoded string, as shown in this answer and this
    answer    ,
    for example:

    base64_string= base64.b64encode(image_bytes).decode('utf-8')
return base64_string

    which can then be used to display the image in an HTML page as shown
    here, for instance:

    <img src="data:image/png;base64,ADD_BASE64_STRING_HERE>
  2. or, send the raw bytes from the server, as you currently do. However, it
    might be best not to use a StreamingResponse (as shown in the
    example you provided) for sending the raw bytes, as, in your case, the
    entire image bytes are already loaded into memory. Thus, you should rather
    use a custom Response, as explained in this answer and this
    answer    , for example:

    return Response(image_bytes, media_type='image/jpeg')

    On client side (using JavaScript), convert the raw bytes into either a
    base64-encoded string (using btoa(), String.fromCharCode()
    and Uint8Array), or a Blob object (and then call
    URL.createObjectURL() to create a URL representing the Blob
    object).

The examples below demonstrate how to implement the two methods explained in Option 2 above (i.e., having the server sending raw image bytes and, on client side, converting the image bytes to either a base64-encoded string or Blob object), using either Fetch API or Axios library.

Using Fetch API

Option 1 – Convert raw image bytes into Blob object

fetch('/predict', {
        method: 'POST',
        body: formData,
    })
    .then(response => response.blob())
    .then(blob => {
        var blobURL = URL.createObjectURL(blob);
        var image = document.getElementById("myImage");
        image.onload = function(){
            URL.revokeObjectURL(this.src); // release the blob URL once the image is loaded
        }
        image.src = blobURL;
    })
    .catch(error => {
        console.error(error);
    });

Option 2 – Convert raw image bytes into base64-encoded string

fetch('/predict', {
        method: 'POST',
        body: formData,
    })
    .then(response => {
        contentType = response.headers.get('content-type')
        return response.arrayBuffer();
    })
    .then(arrayBuffer => {
        base64string = btoa(String.fromCharCode(...new Uint8Array(arrayBuffer)))
        var image = document.getElementById("myImage");
        image.src = "data:" + contentType + ";base64," + base64string;
    })
    .catch(error => {
        console.error(error);
    });

Remember to define an <img> tag in your HTML file, where you wish to display the image:

<img id="myImage" src="">

Using Axios

Option 1 – Convert raw image bytes into Blob object

axios({
        method: 'POST',
        url: '/upload',
        data: formData,
        headers: {
            'Content-Type': 'multipart/form-data'
        },
        responseType: "blob"
    })
    .then(response => {
        var blobURL = URL.createObjectURL(response.data);
        var image = document.getElementById("myImage");
        image.onload = function(){
            URL.revokeObjectURL(this.src); // release the blob URL once the image is loaded
        }
        image.src = blobURL;
    })
    .catch(error => {
        console.error(error);
    });

Option 2 – Convert raw image bytes into base64-encoded string

axios({
        method: 'POST',
        url: '/predict',
        data: formData,
        headers: {
            'Content-Type': 'multipart/form-data'
        },
        responseType: "arraybuffer"
    })
    .then(response => {
        base64string = btoa(String.fromCharCode(...new Uint8Array(response.data)))
        contentType = response.headers['content-type']
        return base64string;
    })
    .then(base64string => {
        var image = document.getElementById("myImage");
        image.src = "data:" + contentType + ";base64," + base64string;
    })
    .catch(error => {
        console.error(error);
    });

Remember to define an <img> tag in your HTML file, where you wish to display the image:

<img id="myImage" src="">
Answered By: Chris