Request.get suddenly started hitting "Temporary failure in name resolution" error
Question:
I have a script that scrapes new chapters from a specific novel on ncode.syosetu.com every week, I need the text for tts, but this last week I suddenly started getting an error from it when it runs.
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='ncode.syosetu.com', port=443): Max retries exceeded with url: /n6371es/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f21b8d64f28>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))
I can access the site in Chrome but ping and tracert get "Request timed out", tracert gets the first 9 or so hops but then it only gets "Request timed out" for the rest.
I also tried using https://www.site24x7.com/ping-test.html to ping the site and it only once got a response on the Johannesburg – ZA location.
I tried adding a User-Agent to the request.get call in a header but that didn’t help. I have also tried different VPN locations, tuning off the VPN, and pausing my firewall but none helped.
I don’t know if it is related but I did have an issue with my internet connection recently, but all I did to fix it was disable then enable my ethernet in Network Connections then uninstall and re-install my VPN.
I have read about the /etc/resolv.conf but I am assuming that would not affect the windows command prompt’s ability to ping the site. But just for reference all that is in the file is the WSL comments on how to stop the auto generation.
Is there some way to be able to get requests to the site working again or some other way to get the text from a specific page that would work?
I’d also just kind of want to know if anyone else is able to ping ncode.syosetu.com successfully?
Update:
I have tried to run a simple interactive python3 requests.get in WSL to google.ca and am hitting the same error. I also tried the same command on my work computer, but to ncode.syosetu.com, and it worked there. So I guess the issue on with my computer/internet. Though even on my work computer I could not ping or tracert ncode.syosetu.com.
I also tried:
s = requests.Session()
s.mount('http://', requests.adapters.HTTPAdapter(max_retries=1000))
s.get('https://ncode.syosetu.com/n6371es/')
but still got the Max retries exceeded with url (Caused by Temporary failure in name resolution)
Answers:
In the end the issue was with WSL, for some reason it seemed to have lost the ability to access the internet.
I found the fix in https://github.com/microsoft/WSL/issues/5420.
The fix was to:
sudo bash -c 'echo "[network]" > /etc/wsl.conf'
sudo bash -c 'echo "generateResolvConf = false" >> /etc/wsl.conf'
(in powershell) wsl --shutdown
sudo rm /etc/resolv.conf
sudo bash -c 'echo "nameserver 8.8.8.8" > /etc/resolv.conf'
This basically told WSL to stop regenerating the /etc/resolv.conf, restart WSL to apply the /etc/wsl.conf changes, remove the dead link /etc/resolv.conf, then update the DNS with a new /etc/resolv.conf
I have a script that scrapes new chapters from a specific novel on ncode.syosetu.com every week, I need the text for tts, but this last week I suddenly started getting an error from it when it runs.
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='ncode.syosetu.com', port=443): Max retries exceeded with url: /n6371es/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f21b8d64f28>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution',))
I can access the site in Chrome but ping and tracert get "Request timed out", tracert gets the first 9 or so hops but then it only gets "Request timed out" for the rest.
I also tried using https://www.site24x7.com/ping-test.html to ping the site and it only once got a response on the Johannesburg – ZA location.
I tried adding a User-Agent to the request.get call in a header but that didn’t help. I have also tried different VPN locations, tuning off the VPN, and pausing my firewall but none helped.
I don’t know if it is related but I did have an issue with my internet connection recently, but all I did to fix it was disable then enable my ethernet in Network Connections then uninstall and re-install my VPN.
I have read about the /etc/resolv.conf but I am assuming that would not affect the windows command prompt’s ability to ping the site. But just for reference all that is in the file is the WSL comments on how to stop the auto generation.
Is there some way to be able to get requests to the site working again or some other way to get the text from a specific page that would work?
I’d also just kind of want to know if anyone else is able to ping ncode.syosetu.com successfully?
Update:
I have tried to run a simple interactive python3 requests.get in WSL to google.ca and am hitting the same error. I also tried the same command on my work computer, but to ncode.syosetu.com, and it worked there. So I guess the issue on with my computer/internet. Though even on my work computer I could not ping or tracert ncode.syosetu.com.
I also tried:
s = requests.Session()
s.mount('http://', requests.adapters.HTTPAdapter(max_retries=1000))
s.get('https://ncode.syosetu.com/n6371es/')
but still got the Max retries exceeded with url (Caused by Temporary failure in name resolution)
In the end the issue was with WSL, for some reason it seemed to have lost the ability to access the internet.
I found the fix in https://github.com/microsoft/WSL/issues/5420.
The fix was to:
sudo bash -c 'echo "[network]" > /etc/wsl.conf'
sudo bash -c 'echo "generateResolvConf = false" >> /etc/wsl.conf'
(in powershell) wsl --shutdown
sudo rm /etc/resolv.conf
sudo bash -c 'echo "nameserver 8.8.8.8" > /etc/resolv.conf'
This basically told WSL to stop regenerating the /etc/resolv.conf, restart WSL to apply the /etc/wsl.conf changes, remove the dead link /etc/resolv.conf, then update the DNS with a new /etc/resolv.conf