Dynamically display incoming HTTP request data on a Python/Flask page without reload
Question:
I’m trying to build a quick and simple webpage to test an HTTP Requests service that I’m building, and Flask seems to be making it way harder than it should be. All I want is to display any incoming HTTP Requests on the page, and then return the received payload to the service that called the webpage.
Copying the Flask tutorial (plus the code for collecting HTTP requests) left me with this app.py file:
from flask import Flask, request, abort, jsonify, render_template, Response, stream_with_context, url_for, redirect
app = Flask(__name__, template_folder = '')
responses = []
@app.route('/')
def index():
return render_template('index.html', resps = responses)
@app.route('/request', methods = ['GET', 'POST'])
def add_message():
print("Request dictionary: {}".format(request.json))
responses.append(request.json)
return redirect('')
app.run(host = '0.0.0.0', port = 81, debug = True)
I understand that request.json does not actually display all of the incoming data, but that is easily solvable assuming that the dynamic display of requests is at all possible.
And here’s my index.html:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Flask Response Test Page</title>
</head>
<body>
Web App with Python Flask!<br><br>
<label id="value_lable">
{% for resp in resps %}
{{ resp }}<br>
{% endfor %}
</label>
</body>
</html>
My service that sends Python requests is really simple and works great. Basically, users type in the endpoint, payload, and output variable, and my service will send the request to endpoint and store the response in variable. I know it works because I can reload my Flask test page and it will show the list of received requests, and my service is printing out the input JSON as the request response. This code runs inside a Docker container so I am using http://host.docker.internal:81/request as my endpoint for the purposes of this test as per this answer.
import requests
class JSONGETRequest():
def on_start(self, executive):
# dict.get() returns None if the dict doesn't have the requested key
endpoint = self.node.data.get("url")
payload = self.node.data.get("payload")
variable = self.node.data.get("variable")
out = requests.get(endpoint, json = payload, headers = {"User-Agent": ""}).json()
executive.set_context(variable, out)
executive.next(self.node, "output_1")
TL;DR:
I’ve tried a bunch of different answers posted on StackOverflow that are related to this question and none of them worked. Everything after this is the various things I tried already.
I attempted to use this page (Flask Dynamic data update without reload page) but that uses a button to force the page to reload, and I cannot find anything that indicates AJAX can activate/run a function whenever the page receives a Request. The vast majority of questions I’ve found relating to "flask update page data without refresh" all use a button but I really just want a streamed output of the requests as they come in.
Edit: Based on NoCommandLine’s comment I was able to fix my implementation of this answer, but it still fails to update the page in real-time, and requires a page reload in order to update the displayed data.
I tried to use this answer ~~but got the error werkzeug.routing.exceptions.BuildError: Could not build url for endpoint 'request'. Did you mean 'static' instead? when using~~ with the following code:
@app.route('/requests', methods = ['GET', 'POST'])
def add_message():
print("Request dictionary: {}".format(request.json))
responses.append(request.json)
def inner():
# simulate a long process to watch
for i in responses:
# this value should be inserted into an HTML template
yield i + '<br/>n'
return Response(inner(), mimetype = 'text/html')
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Flask Response Test Page</title>
</head>
<body>
Web App with Python Flask!<br><br>
<div>
<!-- url_for() is based off of the target method name, not the page name-->
<iframe frameborder="0"
onresize="noresize"
src="{{ url_for('add_message')}}"
style='background: transparent; width: 100%; height:100%;'>
</iframe>
</div>
</body>
</html>
~~Which honestly makes no sense since the request page should exist since I created the @app.route('/request')– it fails with both url_for('request') and url_for('/request').~~
Supposedly there is "quite a few tutorials" on how to use AJAX but not a single on that I’ve found that even MENTIONS HTTP requests.
How is there not a website or service that already does this? Just take in any incoming requests and print them out. There’s no reason this should be this hard.
I tried using @app.after_request but it seems to be useless. And for some reason render_template is just a completely useless function that does absolutely nothing helpful:
@app.route('/request', methods = ['GET', 'POST'])
def add_message():
print("Request dictionary: {}".format(request.json))
responses.append(request.json)
return request.json
@app.after_request
def update(response):
render_template('index.html', resps = responses)
return response
This question is basically a duplicate but has no (useful) answers.
Answers:
Try using flask_socketio. This will allow you to send messages between the front and back end. I’m not sure this is the best solution but here is how it works.
from flask_socketio import SocketIO
from flask import Flask
app = Flask(__name__)
socketio = SocketIO()
socketio.init_app(app)
@socketio.on('message')
def handleMessage(msg):
print(f"Message: {msg}")
send(msg, broadcast=True)
if __name__ == "__main__":
socketio.run(
app, host='0.0.0.0', port="5000", debug=True)
Now on the front end use js to send data
$(document).ready(function() {
var socket = io.connect(...);
socket.on('connect', function() {
socket.emit('my event', ... , function(data){
console.log(data);
});
});
socket.on('my response', function(data){
console.log(data);
});
});
If you face a multi-theading, instantiate socketid with threading enabled
socketio = SocketIO(app, async_mode="threading")
I hope i was able to help.
Three days later and I’ve finally figured out the solution, based somewhat on the Flask_SocketIO idea that @ElieSaad posted about in their answer.
The backend.
- This question answers how to access (and therefore return) the incoming request information from the
@app.after_request-tagged method.
from flask import Flask, request, render_template
from flask_socketio import SocketIO
app = Flask(__name__, template_folder = '')
socketio = SocketIO(app)
socketio.init_app(app)
responses = [{''}]
# Step 1: Load the base webpage
@app.route('/')
def index():
return render_template('index.html', resps = responses)
# Step 2: Receive incoming requests
@app.route('/requests', methods = ['GET', 'POST'])
def add_message():
incoming = request.get_json(silent = True)
if incoming:
print(f"Request dictionary: "{request.json}"")
responses.append(request.json)
else:
print(f"Request: {request}")
return incoming
# Step 3: Send incoming request information to the frontend, and return the request back to the source
@app.after_request
def output(response):
print(f"emitting received request...'")
socketio.emit('my_response', request.get_json(silent = True), broadcast = True, namespace = '/')
return response
# This just runs the app
socketio.run(app, host = '0.0.0.0', port = "81", debug = True, allow_unsafe_werkzeug = True)
The frontend.
- This question answers how to continuously add information to an existing HTML tag.
- This question was more information on Flask Sockets and also got me on the right track of using the socket methods correctly and modifying the HTML code with them.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Flask Response Test Page</title>
</head>
<body>
Web App with Python Flask!<br><br>
<div id="place_for_responses"></div>
<script crossorigin="anonymous"
integrity="sha512-q/dWJ3kcmjBLU4Qc47E4A9kTB4m3wuTY7vkFJDTZKjTs8jhyGQnaUrxa0Ytd0ssMZhbNua9hE+E7Qv1j+DyZwA=="
src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.0.1/socket.io.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script charset="utf-8" type="text/javascript">
var socket = io.connect('http://' + document.domain + ':' + location.port);
socket.on('connect', function() {
console.log("Yay! I'm connected!");
socket.emit('my_event', 'I'm connected!');
});
// This function receives information from the backend and updates the HTML
socket.on('my_response', function(data) {
console.log('[EVENT CALLED] my_response')
let htmlFrag = "<p>" + data + "</p>"
var d1 = document.getElementById('place_for_responses');
d1.insertAdjacentHTML('beforeend', htmlFrag);
socket.emit('my_event', 'Loaded HTML information');
});
</script>
</body>
</html>
The other answer put me on the right track with using sockets to transmit data between the frontend and backend, but had none of the actual information necessary to make the webpage dynamic loading happen. Hopefully this can be of great help to future coders who also couldn’t find any of the necessary information to write what should have been an extremely simple program.
P.S. There is likely still some extraneous code but it took combining and testing a variety of solutions from like 30 other StackOverflow posts and this works so it’s what I’m posting. Feel free to comment ways that this answer could be improved
I’m trying to build a quick and simple webpage to test an HTTP Requests service that I’m building, and Flask seems to be making it way harder than it should be. All I want is to display any incoming HTTP Requests on the page, and then return the received payload to the service that called the webpage.
Copying the Flask tutorial (plus the code for collecting HTTP requests) left me with this app.py file:
from flask import Flask, request, abort, jsonify, render_template, Response, stream_with_context, url_for, redirect
app = Flask(__name__, template_folder = '')
responses = []
@app.route('/')
def index():
return render_template('index.html', resps = responses)
@app.route('/request', methods = ['GET', 'POST'])
def add_message():
print("Request dictionary: {}".format(request.json))
responses.append(request.json)
return redirect('')
app.run(host = '0.0.0.0', port = 81, debug = True)
I understand that request.json does not actually display all of the incoming data, but that is easily solvable assuming that the dynamic display of requests is at all possible.
And here’s my index.html:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Flask Response Test Page</title>
</head>
<body>
Web App with Python Flask!<br><br>
<label id="value_lable">
{% for resp in resps %}
{{ resp }}<br>
{% endfor %}
</label>
</body>
</html>
My service that sends Python requests is really simple and works great. Basically, users type in the endpoint, payload, and output variable, and my service will send the request to endpoint and store the response in variable. I know it works because I can reload my Flask test page and it will show the list of received requests, and my service is printing out the input JSON as the request response. This code runs inside a Docker container so I am using http://host.docker.internal:81/request as my endpoint for the purposes of this test as per this answer.
import requests
class JSONGETRequest():
def on_start(self, executive):
# dict.get() returns None if the dict doesn't have the requested key
endpoint = self.node.data.get("url")
payload = self.node.data.get("payload")
variable = self.node.data.get("variable")
out = requests.get(endpoint, json = payload, headers = {"User-Agent": ""}).json()
executive.set_context(variable, out)
executive.next(self.node, "output_1")
TL;DR:
I’ve tried a bunch of different answers posted on StackOverflow that are related to this question and none of them worked. Everything after this is the various things I tried already.
I attempted to use this page (Flask Dynamic data update without reload page) but that uses a button to force the page to reload, and I cannot find anything that indicates AJAX can activate/run a function whenever the page receives a Request. The vast majority of questions I’ve found relating to "flask update page data without refresh" all use a button but I really just want a streamed output of the requests as they come in.
Edit: Based on NoCommandLine’s comment I was able to fix my implementation of this answer, but it still fails to update the page in real-time, and requires a page reload in order to update the displayed data.
I tried to use this answer ~~but got the error werkzeug.routing.exceptions.BuildError: Could not build url for endpoint 'request'. Did you mean 'static' instead? when using~~ with the following code:
@app.route('/requests', methods = ['GET', 'POST'])
def add_message():
print("Request dictionary: {}".format(request.json))
responses.append(request.json)
def inner():
# simulate a long process to watch
for i in responses:
# this value should be inserted into an HTML template
yield i + '<br/>n'
return Response(inner(), mimetype = 'text/html')
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Flask Response Test Page</title>
</head>
<body>
Web App with Python Flask!<br><br>
<div>
<!-- url_for() is based off of the target method name, not the page name-->
<iframe frameborder="0"
onresize="noresize"
src="{{ url_for('add_message')}}"
style='background: transparent; width: 100%; height:100%;'>
</iframe>
</div>
</body>
</html>
~~Which honestly makes no sense since the request page should exist since I created the @app.route('/request')– it fails with both url_for('request') and url_for('/request').~~
Supposedly there is "quite a few tutorials" on how to use AJAX but not a single on that I’ve found that even MENTIONS HTTP requests.
How is there not a website or service that already does this? Just take in any incoming requests and print them out. There’s no reason this should be this hard.
I tried using @app.after_request but it seems to be useless. And for some reason render_template is just a completely useless function that does absolutely nothing helpful:
@app.route('/request', methods = ['GET', 'POST'])
def add_message():
print("Request dictionary: {}".format(request.json))
responses.append(request.json)
return request.json
@app.after_request
def update(response):
render_template('index.html', resps = responses)
return response
This question is basically a duplicate but has no (useful) answers.
Try using flask_socketio. This will allow you to send messages between the front and back end. I’m not sure this is the best solution but here is how it works.
from flask_socketio import SocketIO
from flask import Flask
app = Flask(__name__)
socketio = SocketIO()
socketio.init_app(app)
@socketio.on('message')
def handleMessage(msg):
print(f"Message: {msg}")
send(msg, broadcast=True)
if __name__ == "__main__":
socketio.run(
app, host='0.0.0.0', port="5000", debug=True)
Now on the front end use js to send data
$(document).ready(function() {
var socket = io.connect(...);
socket.on('connect', function() {
socket.emit('my event', ... , function(data){
console.log(data);
});
});
socket.on('my response', function(data){
console.log(data);
});
});
If you face a multi-theading, instantiate socketid with threading enabled
socketio = SocketIO(app, async_mode="threading")
I hope i was able to help.
Three days later and I’ve finally figured out the solution, based somewhat on the Flask_SocketIO idea that @ElieSaad posted about in their answer.
The backend.
- This question answers how to access (and therefore return) the incoming request information from the
@app.after_request-tagged method.
from flask import Flask, request, render_template
from flask_socketio import SocketIO
app = Flask(__name__, template_folder = '')
socketio = SocketIO(app)
socketio.init_app(app)
responses = [{''}]
# Step 1: Load the base webpage
@app.route('/')
def index():
return render_template('index.html', resps = responses)
# Step 2: Receive incoming requests
@app.route('/requests', methods = ['GET', 'POST'])
def add_message():
incoming = request.get_json(silent = True)
if incoming:
print(f"Request dictionary: "{request.json}"")
responses.append(request.json)
else:
print(f"Request: {request}")
return incoming
# Step 3: Send incoming request information to the frontend, and return the request back to the source
@app.after_request
def output(response):
print(f"emitting received request...'")
socketio.emit('my_response', request.get_json(silent = True), broadcast = True, namespace = '/')
return response
# This just runs the app
socketio.run(app, host = '0.0.0.0', port = "81", debug = True, allow_unsafe_werkzeug = True)
The frontend.
- This question answers how to continuously add information to an existing HTML tag.
- This question was more information on Flask Sockets and also got me on the right track of using the socket methods correctly and modifying the HTML code with them.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Flask Response Test Page</title>
</head>
<body>
Web App with Python Flask!<br><br>
<div id="place_for_responses"></div>
<script crossorigin="anonymous"
integrity="sha512-q/dWJ3kcmjBLU4Qc47E4A9kTB4m3wuTY7vkFJDTZKjTs8jhyGQnaUrxa0Ytd0ssMZhbNua9hE+E7Qv1j+DyZwA=="
src="https://cdnjs.cloudflare.com/ajax/libs/socket.io/4.0.1/socket.io.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script charset="utf-8" type="text/javascript">
var socket = io.connect('http://' + document.domain + ':' + location.port);
socket.on('connect', function() {
console.log("Yay! I'm connected!");
socket.emit('my_event', 'I'm connected!');
});
// This function receives information from the backend and updates the HTML
socket.on('my_response', function(data) {
console.log('[EVENT CALLED] my_response')
let htmlFrag = "<p>" + data + "</p>"
var d1 = document.getElementById('place_for_responses');
d1.insertAdjacentHTML('beforeend', htmlFrag);
socket.emit('my_event', 'Loaded HTML information');
});
</script>
</body>
</html>
The other answer put me on the right track with using sockets to transmit data between the frontend and backend, but had none of the actual information necessary to make the webpage dynamic loading happen. Hopefully this can be of great help to future coders who also couldn’t find any of the necessary information to write what should have been an extremely simple program.
P.S. There is likely still some extraneous code but it took combining and testing a variety of solutions from like 30 other StackOverflow posts and this works so it’s what I’m posting. Feel free to comment ways that this answer could be improved