Issue with Websockets (WSS) server implemented in Python
Question:
I am aware of the browser-based implementation at websockets.readthedocs.io as well as the secure implementation(wss) but I am having extreme difficulty getting a mixture of both working(Browser-based WSS).
Currently, I’m testing the socket server on two different environments, windows 10 NT (latest) over ws & CentOS 7.9 over wss and behind a Cloudflare proxy. The windows tests are working just fine but when trying the CentOS 7 implementation in a browser the socket fails and is throwing closing instantly for an unknown reason (Client-side Websockets Errors, eventPhase #2) in my websocket.onerror callback.
I suspect there is an issue with my sslContext being incorrect and should admit that I don’t have much experience with certificates at a development level. CORS won’t allow the handshake to complete with a certificate that doesn’t match the initial page that’s loading the websocket so I’ve installed a Cloudflare origin certificate along with enabling authenticated origin pulls in the CF dashboard. Since visiting the page that initializes the websocket, I have confirmed that my origin cert is being served correctly to the browser but am not sure that my websocket is serving the cert to the CF edge server as it should, which is causing an error with CF.
After reading many threads on CF’s forums I am left with the idea that CF can proxy WSS just fine as long as things are correct on the origin server.
I am attempting the websocket on port 8443 which CF support suggested.
I suspect that I’m incorrectly using the provided Cloudflare origin cert with the sslContext.
In reference to the sslContext below, the cert.crt is the CA and the cert.key is the corresponding key, both of which I received when I generated the CF origin cert. Is this correct and if not, how can use a CF origin cert to secure the websocket?
Here is the relevant sslContext:
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_context.load_cert_chain("/var/www/mysite.org/cert/cert.crt", "/var/www/mysite.org/cert/cert.key")
Suspecting that this is only a browser-based issue, I used a modified version of the browser-based example on ReadTheDocs (client) to augment a loosely-secured browser, sure enough, after correctly using Certifi to create an sslContext[2nd paste below], I was met with the exception:
Traceback (most recent call last):
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 87, in _run_code
exec(code, run_globals)
File "c:UsersUser.vscodeextensionsms-python.python-2021.5.829140558pythonFileslibpythondebugpy__main__.py", line 45, in <module>
cli.main()
File "c:UsersUser.vscodeextensionsms-python.python-2021.5.829140558pythonFileslibpythondebugpy/..debugpyservercli.py", line 444, in main
run()
File "c:UsersUser.vscodeextensionsms-python.python-2021.5.829140558pythonFileslibpythondebugpy/..debugpyservercli.py", line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 87, in _run_code
exec(code, run_globals)
File "c:UsersUserDesktopmysitesocketclient.py", line 23, in <module>
asyncio.get_event_loop().run_until_complete(test())
File "C:UsersUserAppDataLocalProgramsPythonPython39libasynciobase_events.py", line 642, in run_until_complete
return future.result()
File "c:UsersUserDesktopmysitesocketclient.py", line 13, in test
async with websockets.connect(
File "C:UsersUserAppDataLocalProgramsPythonPython39libsite-packageswebsocketslegacyclient.py", line 604, in __aenter__
return await self
File "C:UsersUserAppDataLocalProgramsPythonPython39libsite-packageswebsocketslegacyclient.py", line 629, in __await_impl__
await protocol.handshake(
File "C:UsersUserAppDataLocalProgramsPythonPython39libsite-packageswebsocketslegacyclient.py", line 388, in handshake
raise InvalidStatusCode(status_code)
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 525
Here is the test client that returned the 525 response code:
import asyncio
import ssl
import websockets
import certifi
import logging
logging.basicConfig(filename="client.log", level=logging.DEBUG)
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ssl_context.load_verify_locations(certifi.where())
async def test():
uri = "wss://mysite.org:8443/"
async with websockets.connect(
uri, ssl=ssl_context
) as websocket:
await websocket.send("test")
print("> test")
response = await websocket.recv()
print(f"< {response}")
asyncio.get_event_loop().run_until_complete(test())
Here is all of the relevant script from the server side:
if 'nt' in name:
ENUMS.PLATFORM = 0
platform = DEBUG_WINDOWS_NT
start_server = websockets.serve(
server,
platform.adapter,
platform.port,
max_size=9000000
)
else:
ENUMS.PLATFORM = 1
platform = PRODUCTION_CENTOS_7_xx
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_context.load_cert_chain("/var/www/mysite.org/cert/cert.crt", "/var/www/mysite.org/cert/cert.key")
start_server = websockets.serve(
server,
platform.adapter,#0.0.0.0
platform.port,#8443
max_size=9000000,
ssl=ssl_context
)
try:
asyncio.get_event_loop().run_until_complete(start_server)
print("Mysite Socket -> LISTENING ON %s:%s" % (platform.adapter, platform.port,))
asyncio.get_event_loop().run_forever()
except Exception as e:
print(e)
And here is the corresponding Client side JS (if useful):
let sAddress;
if (location.hostname.includes('mysite.org')){
sAddress = `wss://${location.hostname}:8443/`
}else{
sAddress = `ws://${location.hostname}:5567/`
}
console.log(`URI: ${sAddress}`);
methods = [
{
action: "detection",
captcha: "",
blob: undefined
},
];
this.websocket = new WebSocket(sAddress);
this.websocket.onerror = function (event){
console.log(event);
switch (event.eventPhase) {
case 2:
alert("The connection is going through the closing handshake, or the close() method has been invoked.");
break;
default:
alert("Unknown error.");
break;
}
location.reload();
}
Here is the logging output from my server script after attempting to connect (both client and browser had same results):
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
Note the "state = CONNECTING" events, they continue, constantly..
So essentially a connection is being made, the client side closes but CF keeps sending requests to the origin server.
After seeing this I decided to capture the traffic on port 8443, here is a dump of a single attempt from the client to the socket server, this reoccurs constantly in the same order after one attempt is made.
Upon inspecting the dump in a hex editor, I can see that the CF origin cert is in the packet that has a length of 1662.
From my best knowledge, the handshake is in fact occurring through the proxy but fails/is dropped by the browser which leaves the CF edge server retrying the handshake, maybe expecting a close header or something that it’s not getting from the client.
Either way, it seems like the handshake is being dropped by the browser/client in a facultative way.
Tcpdump of traffic on port 8443 during browser/client attempt
01:04:21.585671 IP CLOUDFLARE > MY_SERVER: Flags [.], ack 1, win 64, length 0
01:04:21.586293 IP CLOUDFLARE > MY_SERVER: Flags [P.], seq 1:227, ack 1, win 64, length 226
01:04:21.586321 IP MY_SERVER > CLOUDFLARE: Flags [.], ack 227, win 237, length 0
01:04:21.590265 IP MY_SERVER > CLOUDFLARE: Flags [P.], seq 1:1663, ack 227, win 237, length 1662
01:04:21.749807 IP CLOUDFLARE > MY_SERVER: Flags [.], ack 1461, win 67, length 0
01:04:21.749871 IP CLOUDFLARE > MY_SERVER: Flags [.], ack 1663, win 70, length 0
01:04:21.751082 IP CLOUDFLARE > MY_SERVER: Flags [P.], seq 227:1583, ack 1663, win 70, length 1356
01:04:21.751978 IP MY_SERVER > CLOUDFLARE: Flags [F.], seq 1663, ack 1583, win 258, length 0
01:04:21.911537 IP CLOUDFLARE > MY_SERVER: Flags [F.], seq 1583, ack 1664, win 70, length 0
01:04:21.911639 IP MY_SERVER > CLOUDFLARE: Flags [.], ack 1584, win 258, length 0
This is where it gets weird…
Finally, I tried bypassing CF and accessing my server directly with the python client:
I get the following exception, with the same DEBUG:websockets.protocol:server - state = CONNECTING message in the log:
Exception has occurred: SSLCertVerificationError
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
File "C:UsersUserDesktopmysite.orgsocketclient.py", line 13, in test
async with websockets.connect(
File "C:UsersUserDesktopmysite.orgsocketclient.py", line 23, in <module>
asyncio.get_event_loop().run_until_complete(test())
Here is the Tcpdump from attempting direct access to the server from the Python client:
07:10:18.135487 IP MYIP > MYSERVER: Flags [S], seq 4035557944, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
07:10:18.135641 IP MYSERVER > MYIP: Flags [S.], seq 4244322261, ack 4035557945, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
07:10:18.306419 IP MYIP > MYSERVER: Flags [.], ack 1, win 1026, length 0
07:10:18.312218 IP MYIP > MYSERVER: Flags [P.], seq 1:518, ack 1, win 1026, length 517
07:10:18.312315 IP MYSERVER > MYIP: Flags [.], ack 518, win 237, length 0
07:10:18.316156 IP MYSERVER > MYIP: Flags [P.], seq 1:1663, ack 518, win 237, length 1662
07:10:18.485326 IP MYIP > MYSERVER: Flags [.], ack 1663, win 1026, length 0
07:10:18.485386 IP MYIP > MYSERVER: Flags [F.], seq 518, ack 1663, win 1026, length 0
07:10:18.485978 IP MYSERVER > MYIP: Flags [F.], seq 1663, ack 519, win 237, length 0
07:10:18.657095 IP MYIP > MYSERVER: Flags [.], ack 1664, win 1026, length 0
TLDR:
- Is my sslContext correct?
- Is my use of the CF origin cert correct?
- Is this even possible and if so, what am I doing wrong?
At this point I am stumped and a bit overwhelmed, a truly unexpected speedbump in an otherwise too-good-to-be-true framework(websockets <3).
Please let me know if further information is needed and I do apologize if the question is a bit long/messy.
Any feedback or solutions will be greatly appreciated as I want our visitors to have a secure experience, thanks!
Answers:
I’ve resolved this issue!
I was using the cert incorrectly and didn’t need to use an origin cert/authed origin pulls to get Cloudflare to proxy the client in the first place.
My resolution was creating a self-signed cert with the hostname *.mysite.org, CF proxies it no problem 🙂
Create a self-signed cert
openssl req -new -x509 -days 365 -nodes -out cert.pem -keyout cert.pem
Correctly present the self-signed cert to the CF edge server
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_context.load_cert_chain("cert.pem", "cert.pem")
start_server = websockets.serve(
server,
platform.adapter,
platform.port,
max_size=9000000,
ssl=ssl_context
)
I am aware of the browser-based implementation at websockets.readthedocs.io as well as the secure implementation(wss) but I am having extreme difficulty getting a mixture of both working(Browser-based WSS).
Currently, I’m testing the socket server on two different environments, windows 10 NT (latest) over ws & CentOS 7.9 over wss and behind a Cloudflare proxy. The windows tests are working just fine but when trying the CentOS 7 implementation in a browser the socket fails and is throwing closing instantly for an unknown reason (Client-side Websockets Errors, eventPhase #2) in my websocket.onerror callback.
I suspect there is an issue with my sslContext being incorrect and should admit that I don’t have much experience with certificates at a development level. CORS won’t allow the handshake to complete with a certificate that doesn’t match the initial page that’s loading the websocket so I’ve installed a Cloudflare origin certificate along with enabling authenticated origin pulls in the CF dashboard. Since visiting the page that initializes the websocket, I have confirmed that my origin cert is being served correctly to the browser but am not sure that my websocket is serving the cert to the CF edge server as it should, which is causing an error with CF.
After reading many threads on CF’s forums I am left with the idea that CF can proxy WSS just fine as long as things are correct on the origin server.
I am attempting the websocket on port 8443 which CF support suggested.
I suspect that I’m incorrectly using the provided Cloudflare origin cert with the sslContext.
In reference to the sslContext below, the cert.crt is the CA and the cert.key is the corresponding key, both of which I received when I generated the CF origin cert. Is this correct and if not, how can use a CF origin cert to secure the websocket?
Here is the relevant sslContext:
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_context.load_cert_chain("/var/www/mysite.org/cert/cert.crt", "/var/www/mysite.org/cert/cert.key")
Suspecting that this is only a browser-based issue, I used a modified version of the browser-based example on ReadTheDocs (client) to augment a loosely-secured browser, sure enough, after correctly using Certifi to create an sslContext[2nd paste below], I was met with the exception:
Traceback (most recent call last):
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 87, in _run_code
exec(code, run_globals)
File "c:UsersUser.vscodeextensionsms-python.python-2021.5.829140558pythonFileslibpythondebugpy__main__.py", line 45, in <module>
cli.main()
File "c:UsersUser.vscodeextensionsms-python.python-2021.5.829140558pythonFileslibpythondebugpy/..debugpyservercli.py", line 444, in main
run()
File "c:UsersUser.vscodeextensionsms-python.python-2021.5.829140558pythonFileslibpythondebugpy/..debugpyservercli.py", line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:UsersUserAppDataLocalProgramsPythonPython39librunpy.py", line 87, in _run_code
exec(code, run_globals)
File "c:UsersUserDesktopmysitesocketclient.py", line 23, in <module>
asyncio.get_event_loop().run_until_complete(test())
File "C:UsersUserAppDataLocalProgramsPythonPython39libasynciobase_events.py", line 642, in run_until_complete
return future.result()
File "c:UsersUserDesktopmysitesocketclient.py", line 13, in test
async with websockets.connect(
File "C:UsersUserAppDataLocalProgramsPythonPython39libsite-packageswebsocketslegacyclient.py", line 604, in __aenter__
return await self
File "C:UsersUserAppDataLocalProgramsPythonPython39libsite-packageswebsocketslegacyclient.py", line 629, in __await_impl__
await protocol.handshake(
File "C:UsersUserAppDataLocalProgramsPythonPython39libsite-packageswebsocketslegacyclient.py", line 388, in handshake
raise InvalidStatusCode(status_code)
websockets.exceptions.InvalidStatusCode: server rejected WebSocket connection: HTTP 525
Here is the test client that returned the 525 response code:
import asyncio
import ssl
import websockets
import certifi
import logging
logging.basicConfig(filename="client.log", level=logging.DEBUG)
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ssl_context.load_verify_locations(certifi.where())
async def test():
uri = "wss://mysite.org:8443/"
async with websockets.connect(
uri, ssl=ssl_context
) as websocket:
await websocket.send("test")
print("> test")
response = await websocket.recv()
print(f"< {response}")
asyncio.get_event_loop().run_until_complete(test())
Here is all of the relevant script from the server side:
if 'nt' in name:
ENUMS.PLATFORM = 0
platform = DEBUG_WINDOWS_NT
start_server = websockets.serve(
server,
platform.adapter,
platform.port,
max_size=9000000
)
else:
ENUMS.PLATFORM = 1
platform = PRODUCTION_CENTOS_7_xx
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_context.load_cert_chain("/var/www/mysite.org/cert/cert.crt", "/var/www/mysite.org/cert/cert.key")
start_server = websockets.serve(
server,
platform.adapter,#0.0.0.0
platform.port,#8443
max_size=9000000,
ssl=ssl_context
)
try:
asyncio.get_event_loop().run_until_complete(start_server)
print("Mysite Socket -> LISTENING ON %s:%s" % (platform.adapter, platform.port,))
asyncio.get_event_loop().run_forever()
except Exception as e:
print(e)
And here is the corresponding Client side JS (if useful):
let sAddress;
if (location.hostname.includes('mysite.org')){
sAddress = `wss://${location.hostname}:8443/`
}else{
sAddress = `ws://${location.hostname}:5567/`
}
console.log(`URI: ${sAddress}`);
methods = [
{
action: "detection",
captcha: "",
blob: undefined
},
];
this.websocket = new WebSocket(sAddress);
this.websocket.onerror = function (event){
console.log(event);
switch (event.eventPhase) {
case 2:
alert("The connection is going through the closing handshake, or the close() method has been invoked.");
break;
default:
alert("Unknown error.");
break;
}
location.reload();
}
Here is the logging output from my server script after attempting to connect (both client and browser had same results):
DEBUG:asyncio:Using selector: EpollSelector
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
DEBUG:websockets.protocol:server - state = CONNECTING
Note the "state = CONNECTING" events, they continue, constantly..
So essentially a connection is being made, the client side closes but CF keeps sending requests to the origin server.
After seeing this I decided to capture the traffic on port 8443, here is a dump of a single attempt from the client to the socket server, this reoccurs constantly in the same order after one attempt is made.
Upon inspecting the dump in a hex editor, I can see that the CF origin cert is in the packet that has a length of 1662.
From my best knowledge, the handshake is in fact occurring through the proxy but fails/is dropped by the browser which leaves the CF edge server retrying the handshake, maybe expecting a close header or something that it’s not getting from the client.
Either way, it seems like the handshake is being dropped by the browser/client in a facultative way.
Tcpdump of traffic on port 8443 during browser/client attempt
01:04:21.585671 IP CLOUDFLARE > MY_SERVER: Flags [.], ack 1, win 64, length 0
01:04:21.586293 IP CLOUDFLARE > MY_SERVER: Flags [P.], seq 1:227, ack 1, win 64, length 226
01:04:21.586321 IP MY_SERVER > CLOUDFLARE: Flags [.], ack 227, win 237, length 0
01:04:21.590265 IP MY_SERVER > CLOUDFLARE: Flags [P.], seq 1:1663, ack 227, win 237, length 1662
01:04:21.749807 IP CLOUDFLARE > MY_SERVER: Flags [.], ack 1461, win 67, length 0
01:04:21.749871 IP CLOUDFLARE > MY_SERVER: Flags [.], ack 1663, win 70, length 0
01:04:21.751082 IP CLOUDFLARE > MY_SERVER: Flags [P.], seq 227:1583, ack 1663, win 70, length 1356
01:04:21.751978 IP MY_SERVER > CLOUDFLARE: Flags [F.], seq 1663, ack 1583, win 258, length 0
01:04:21.911537 IP CLOUDFLARE > MY_SERVER: Flags [F.], seq 1583, ack 1664, win 70, length 0
01:04:21.911639 IP MY_SERVER > CLOUDFLARE: Flags [.], ack 1584, win 258, length 0
This is where it gets weird…
Finally, I tried bypassing CF and accessing my server directly with the python client:
I get the following exception,
with the same DEBUG:websockets.protocol:server - state = CONNECTINGmessage in the log:
Exception has occurred: SSLCertVerificationError
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
File "C:UsersUserDesktopmysite.orgsocketclient.py", line 13, in test
async with websockets.connect(
File "C:UsersUserDesktopmysite.orgsocketclient.py", line 23, in <module>
asyncio.get_event_loop().run_until_complete(test())
Here is the Tcpdump from attempting direct access to the server from the Python client:
07:10:18.135487 IP MYIP > MYSERVER: Flags [S], seq 4035557944, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
07:10:18.135641 IP MYSERVER > MYIP: Flags [S.], seq 4244322261, ack 4035557945, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
07:10:18.306419 IP MYIP > MYSERVER: Flags [.], ack 1, win 1026, length 0
07:10:18.312218 IP MYIP > MYSERVER: Flags [P.], seq 1:518, ack 1, win 1026, length 517
07:10:18.312315 IP MYSERVER > MYIP: Flags [.], ack 518, win 237, length 0
07:10:18.316156 IP MYSERVER > MYIP: Flags [P.], seq 1:1663, ack 518, win 237, length 1662
07:10:18.485326 IP MYIP > MYSERVER: Flags [.], ack 1663, win 1026, length 0
07:10:18.485386 IP MYIP > MYSERVER: Flags [F.], seq 518, ack 1663, win 1026, length 0
07:10:18.485978 IP MYSERVER > MYIP: Flags [F.], seq 1663, ack 519, win 237, length 0
07:10:18.657095 IP MYIP > MYSERVER: Flags [.], ack 1664, win 1026, length 0
TLDR:
- Is my sslContext correct?
- Is my use of the CF origin cert correct?
- Is this even possible and if so, what am I doing wrong?
At this point I am stumped and a bit overwhelmed, a truly unexpected speedbump in an otherwise too-good-to-be-true framework(websockets <3).
Please let me know if further information is needed and I do apologize if the question is a bit long/messy.
Any feedback or solutions will be greatly appreciated as I want our visitors to have a secure experience, thanks!
I’ve resolved this issue!
I was using the cert incorrectly and didn’t need to use an origin cert/authed origin pulls to get Cloudflare to proxy the client in the first place.
My resolution was creating a self-signed cert with the hostname *.mysite.org, CF proxies it no problem 🙂
Create a self-signed cert
openssl req -new -x509 -days 365 -nodes -out cert.pem -keyout cert.pem
Correctly present the self-signed cert to the CF edge server
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ssl_context.load_cert_chain("cert.pem", "cert.pem")
start_server = websockets.serve(
server,
platform.adapter,
platform.port,
max_size=9000000,
ssl=ssl_context
)