Make requests using Python over Tor
Question:
I want to make multiple GET requests using Tor to a webpage. I want to use a different ipaddress for each request.
import socks
import socket
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9150)
socket.socket = socks.socksocket
import requests
print (requests.get('http://icanhazip.com')).content
Using this, I made one request. How can I change the ipaddress to make another?
Answers:
Here is the code you want to use (download the stem package using pip install stem
)
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password='your password set for tor controller port in torrc')
print("Success!")
controller.signal(Signal.NEWNYM)
print("New Tor connection processed")
Good luck and hopefully that works.
There are 2 aspects to your question –
- Making requests using Tor
- Renewing the connection as per requirement (in your case, after every request)
Part 1
The first one is easy to do with the latest (upwards of v2.10.0) requests
library with an additional requirement of requests[socks]
for using the socks proxy.
Installation –
pip install requests[socks]
Basic usage –
import requests
def get_tor_session():
session = requests.session()
# Tor uses the 9050 port as the default socks port
session.proxies = {'http': 'socks5://127.0.0.1:9050',
'https': 'socks5://127.0.0.1:9050'}
return session
# Make a request through the Tor connection
# IP visible through Tor
session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)
# Above should print an IP different than your public IP
# Following prints your normal public IP
print(requests.get("http://httpbin.org/ip").text)
Part 2
To renew the Tor IP, i.e. to have a fresh visible exit IP, you need to be able to connect to the Tor service through it’s ControlPort
and then send a NEWNYM
signal.
Normal Tor installation does not enable the ControlPort
by default. You’ll have to edit your torrc file and uncomment the corresponding lines.
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:05834BCEDD478D1060F1D7E2CE98E9C13075E8D3061D702F63BCD674DE
Please note that the HashedControlPassword
above is for the password "password"
. If you want to set a different password, replace the HashedControlPassword
in the torrc by noting the output from tor --hash-password "<new_password>"
where <new_password>
is the password that you want to set.
……………………………………………………………………..
Warning for Windows users: see post here.
There is an issue on windows where the setting for the controlport in the torrc file is ignored if tor was installed using the following command:
tor --service install
To resolve the issue, after editing your torrc file, type the following commands:
tor --service remove
tor --service install -options ControlPort 9051
……………………………………………………………………..
Okay, so now that we have Tor configured properly, you will have to restart Tor if it is already running.
sudo service tor restart
Tor should now be up & running on the 9051 ControlPort
through which we can send commands to it. I prefer to use the official stem library to control Tor.
Installation –
pip install stem
You may now renew the Tor IP by calling the following function.
Renew IP –
from stem import Signal
from stem.control import Controller
# signal TOR for a new connection
def renew_connection():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="password")
controller.signal(Signal.NEWNYM)
To verify that Tor has a new exit IP, just rerun the code from Part 1. For some reason unknown to me, you need to create a new session
object in order to use the new IP.
session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)
The requests
in requesocks
is super old, it doesn’t have response.json()
and many other stuff.
I would like to keep my code clean. However, requests
currently doesn’t have socks5 supported yet (for more detail, read this thread https://github.com/kennethreitz/requests/pull/478)
So I used Privoxy
as a http proxy that connects Tor for now.
Install and configure Privoxy on Mac
brew install privoxy
vim /usr/local/etc/privoxy/config
# put this line in the config
forward-socks5 / localhost:9050 .
privoxy /usr/local/etc/privoxy/config
Install and configure Privoxy on Ubuntu
sudo apt-get install privoxy
sudo vim /etc/privoxy/config
# put this line in the config
forward-socks5 / localhost:9050 .
sudo /etc/init.d/privoxy restart
Now I can use Tor like a http proxy. Below is my python script.
import requests
proxies = {
'http': 'http://127.0.0.1:8118',
}
print requests.get('http://httpbin.org/ip', proxies=proxies).text
Requests supports proxies using the SOCKS protocol from version 2.10.0.
import requests
proxies = {
'http': 'socks5://localhost:9050',
'https': 'socks5://localhost:9050'
}
url = 'http://httpbin.org/ip'
print(requests.get(url, proxies=proxies).text)
You can use torrequest
library (shameless plug). It’s available on PyPI.
from torrequest import TorRequest
with TorRequest() as tr:
response = tr.get('http://ipecho.net/plain')
print(response.text) # not your IP address
tr.reset_identity()
response = tr.get('http://ipecho.net/plain')
print(response.text) # another IP address, not yours
This answer complete the one of Ashish Nitin Patil for windows
(feel free to update this answer)
Part 2
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:05834BCEDD478D1060F1D7E2CE98E9C13075E8D3061D702F63BCD674DE
The HashedControlPassword
above is the password. If you want to set a different password in the console navigate to Tor BrowserBrowserTorBrowserTor
and type the following commands: tor.exe --hash-password password_XYZ | more
). It will give you something like HashedControlPassword 16:54C092A8...
This is your password. Now you can add it to the torrc file (Tor BrowserBrowserTorBrowserDataTortorrc
).
You will need then to restart Tor:
tor --service remove
tor --service install -options ControlPort 9051
To check if that works type netstat -an
you will now see that port 9051 is open.
Notice that tor --service install -...
will create Tor Win32 Service
. For some reason, it seems you have to stop the service to use the browser (run services.msc
)
EDIT: you will find many pieces of information here (About port number & proxy, Tor, Privoxy, Auto switch user agent…).
This code works fine. Using Tor, it changes the IP address after each request.
import time, socks, socket
from urllib2 import urlopen
from stem import Signal
from stem.control import Controller
nbrOfIpAddresses=3
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password = 'my_pwd')
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
for i in range(0, nbrOfIpAddresses):
newIP=urlopen("http://icanhazip.com").read()
print("NewIP Address: %s" % newIP)
controller.signal(Signal.NEWNYM)
if controller.is_newnym_available() == False:
print("Waitting time for Tor to change IP: "+ str(controller.get_newnym_wait()) +" seconds")
time.sleep(controller.get_newnym_wait())
controller.close()
You can try pure-python tor protocol implementation Torpy. No need original Tor client or Stem dependency at all.
$ pip3 install torpy[requests]
...
$ python3.7
>>> from torpy.http.requests import TorRequests
>>> with TorRequests() as tor_requests:
... print("build circuit")
... with tor_requests.get_session() as sess:
... print(sess.get("http://httpbin.org/ip").json())
... print(sess.get("http://httpbin.org/ip").json())
... print("renew circuit")
... with tor_requests.get_session() as sess:
... print(sess.get("http://httpbin.org/ip").json())
... print(sess.get("http://httpbin.org/ip").json())
...
build circuit
{'origin': '23.129.64.190, 23.129.64.190'}
{'origin': '23.129.64.190, 23.129.64.190'}
renew circuit
{'origin': '198.98.50.112, 198.98.50.112'}
{'origin': '198.98.50.112, 198.98.50.112'}
So each time when you getting new session you get new identity (basically you get new circuit with new exit node). See more examples at readme file https://github.com/torpyorg/torpy
A good function to renew your IP. Windows example
def renew_tor_ip():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="aAjkaI19!!laksjd")
controller.signal(Signal.NEWNYM)
Example of usage
import requests
import time
from stem import Signal
from stem.control import Controller
def get_current_ip():
session = requests.session()
# TO Request URL with SOCKS over TOR
session.proxies = {}
session.proxies['http']='socks5h://localhost:9150'
session.proxies['https']='socks5h://localhost:9150'
try:
r = session.get('http://httpbin.org/ip')
except Exception as e:
print(str(e))
else:
return r.text
#16:8EE7AEE3F32EEEEB605C6AA6C47B47808CA6A81FA0D76546ADC05F0F15 to aAjkaI19!!laksjd
#cmd shell "C:UsersArthurDesktopTor BrowserBrowserTorBrowserTortor.exe" --hash-password aAjkaI19!!laksjd | more
#Torcc config
#ControlPort 9051
#HashedControlPassword 16:8EE7AEE3F32EEEEB605C6AA6C47B47808CA6A81FA0D76546ADC05F0F15
def renew_tor_ip():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="aAjkaI19!!laksjd")
controller.signal(Signal.NEWNYM)
for i in range(5):
print(get_current_ip())
renew_tor_ip()
time.sleep(5)
I want to make multiple GET requests using Tor to a webpage. I want to use a different ipaddress for each request.
import socks
import socket
socks.set_default_proxy(socks.SOCKS5, "127.0.0.1", 9150)
socket.socket = socks.socksocket
import requests
print (requests.get('http://icanhazip.com')).content
Using this, I made one request. How can I change the ipaddress to make another?
Here is the code you want to use (download the stem package using pip install stem
)
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password='your password set for tor controller port in torrc')
print("Success!")
controller.signal(Signal.NEWNYM)
print("New Tor connection processed")
Good luck and hopefully that works.
There are 2 aspects to your question –
- Making requests using Tor
- Renewing the connection as per requirement (in your case, after every request)
Part 1
The first one is easy to do with the latest (upwards of v2.10.0) requests
library with an additional requirement of requests[socks]
for using the socks proxy.
Installation –
pip install requests[socks]
Basic usage –
import requests
def get_tor_session():
session = requests.session()
# Tor uses the 9050 port as the default socks port
session.proxies = {'http': 'socks5://127.0.0.1:9050',
'https': 'socks5://127.0.0.1:9050'}
return session
# Make a request through the Tor connection
# IP visible through Tor
session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)
# Above should print an IP different than your public IP
# Following prints your normal public IP
print(requests.get("http://httpbin.org/ip").text)
Part 2
To renew the Tor IP, i.e. to have a fresh visible exit IP, you need to be able to connect to the Tor service through it’s ControlPort
and then send a NEWNYM
signal.
Normal Tor installation does not enable the ControlPort
by default. You’ll have to edit your torrc file and uncomment the corresponding lines.
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:05834BCEDD478D1060F1D7E2CE98E9C13075E8D3061D702F63BCD674DE
Please note that the HashedControlPassword
above is for the password "password"
. If you want to set a different password, replace the HashedControlPassword
in the torrc by noting the output from tor --hash-password "<new_password>"
where <new_password>
is the password that you want to set.
……………………………………………………………………..
Warning for Windows users: see post here.
There is an issue on windows where the setting for the controlport in the torrc file is ignored if tor was installed using the following command:
tor --service install
To resolve the issue, after editing your torrc file, type the following commands:
tor --service remove
tor --service install -options ControlPort 9051
……………………………………………………………………..
Okay, so now that we have Tor configured properly, you will have to restart Tor if it is already running.
sudo service tor restart
Tor should now be up & running on the 9051 ControlPort
through which we can send commands to it. I prefer to use the official stem library to control Tor.
Installation –
pip install stem
You may now renew the Tor IP by calling the following function.
Renew IP –
from stem import Signal
from stem.control import Controller
# signal TOR for a new connection
def renew_connection():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="password")
controller.signal(Signal.NEWNYM)
To verify that Tor has a new exit IP, just rerun the code from Part 1. For some reason unknown to me, you need to create a new session
object in order to use the new IP.
session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)
The requests
in requesocks
is super old, it doesn’t have response.json()
and many other stuff.
I would like to keep my code clean. However, requests
currently doesn’t have socks5 supported yet (for more detail, read this thread https://github.com/kennethreitz/requests/pull/478)
So I used Privoxy
as a http proxy that connects Tor for now.
Install and configure Privoxy on Mac
brew install privoxy
vim /usr/local/etc/privoxy/config
# put this line in the config
forward-socks5 / localhost:9050 .
privoxy /usr/local/etc/privoxy/config
Install and configure Privoxy on Ubuntu
sudo apt-get install privoxy
sudo vim /etc/privoxy/config
# put this line in the config
forward-socks5 / localhost:9050 .
sudo /etc/init.d/privoxy restart
Now I can use Tor like a http proxy. Below is my python script.
import requests
proxies = {
'http': 'http://127.0.0.1:8118',
}
print requests.get('http://httpbin.org/ip', proxies=proxies).text
Requests supports proxies using the SOCKS protocol from version 2.10.0.
import requests
proxies = {
'http': 'socks5://localhost:9050',
'https': 'socks5://localhost:9050'
}
url = 'http://httpbin.org/ip'
print(requests.get(url, proxies=proxies).text)
You can use torrequest
library (shameless plug). It’s available on PyPI.
from torrequest import TorRequest
with TorRequest() as tr:
response = tr.get('http://ipecho.net/plain')
print(response.text) # not your IP address
tr.reset_identity()
response = tr.get('http://ipecho.net/plain')
print(response.text) # another IP address, not yours
This answer complete the one of Ashish Nitin Patil for windows
(feel free to update this answer)
Part 2
ControlPort 9051
## If you enable the controlport, be sure to enable one of these
## authentication methods, to prevent attackers from accessing it.
HashedControlPassword 16:05834BCEDD478D1060F1D7E2CE98E9C13075E8D3061D702F63BCD674DE
The HashedControlPassword
above is the password. If you want to set a different password in the console navigate to Tor BrowserBrowserTorBrowserTor
and type the following commands: tor.exe --hash-password password_XYZ | more
). It will give you something like HashedControlPassword 16:54C092A8...
This is your password. Now you can add it to the torrc file (Tor BrowserBrowserTorBrowserDataTortorrc
).
You will need then to restart Tor:
tor --service remove
tor --service install -options ControlPort 9051
To check if that works type netstat -an
you will now see that port 9051 is open.
Notice that tor --service install -...
will create Tor Win32 Service
. For some reason, it seems you have to stop the service to use the browser (run services.msc
)
EDIT: you will find many pieces of information here (About port number & proxy, Tor, Privoxy, Auto switch user agent…).
This code works fine. Using Tor, it changes the IP address after each request.
import time, socks, socket
from urllib2 import urlopen
from stem import Signal
from stem.control import Controller
nbrOfIpAddresses=3
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password = 'my_pwd')
socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050)
socket.socket = socks.socksocket
for i in range(0, nbrOfIpAddresses):
newIP=urlopen("http://icanhazip.com").read()
print("NewIP Address: %s" % newIP)
controller.signal(Signal.NEWNYM)
if controller.is_newnym_available() == False:
print("Waitting time for Tor to change IP: "+ str(controller.get_newnym_wait()) +" seconds")
time.sleep(controller.get_newnym_wait())
controller.close()
You can try pure-python tor protocol implementation Torpy. No need original Tor client or Stem dependency at all.
$ pip3 install torpy[requests]
...
$ python3.7
>>> from torpy.http.requests import TorRequests
>>> with TorRequests() as tor_requests:
... print("build circuit")
... with tor_requests.get_session() as sess:
... print(sess.get("http://httpbin.org/ip").json())
... print(sess.get("http://httpbin.org/ip").json())
... print("renew circuit")
... with tor_requests.get_session() as sess:
... print(sess.get("http://httpbin.org/ip").json())
... print(sess.get("http://httpbin.org/ip").json())
...
build circuit
{'origin': '23.129.64.190, 23.129.64.190'}
{'origin': '23.129.64.190, 23.129.64.190'}
renew circuit
{'origin': '198.98.50.112, 198.98.50.112'}
{'origin': '198.98.50.112, 198.98.50.112'}
So each time when you getting new session you get new identity (basically you get new circuit with new exit node). See more examples at readme file https://github.com/torpyorg/torpy
A good function to renew your IP. Windows example
def renew_tor_ip():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="aAjkaI19!!laksjd")
controller.signal(Signal.NEWNYM)
Example of usage
import requests
import time
from stem import Signal
from stem.control import Controller
def get_current_ip():
session = requests.session()
# TO Request URL with SOCKS over TOR
session.proxies = {}
session.proxies['http']='socks5h://localhost:9150'
session.proxies['https']='socks5h://localhost:9150'
try:
r = session.get('http://httpbin.org/ip')
except Exception as e:
print(str(e))
else:
return r.text
#16:8EE7AEE3F32EEEEB605C6AA6C47B47808CA6A81FA0D76546ADC05F0F15 to aAjkaI19!!laksjd
#cmd shell "C:UsersArthurDesktopTor BrowserBrowserTorBrowserTortor.exe" --hash-password aAjkaI19!!laksjd | more
#Torcc config
#ControlPort 9051
#HashedControlPassword 16:8EE7AEE3F32EEEEB605C6AA6C47B47808CA6A81FA0D76546ADC05F0F15
def renew_tor_ip():
with Controller.from_port(port = 9051) as controller:
controller.authenticate(password="aAjkaI19!!laksjd")
controller.signal(Signal.NEWNYM)
for i in range(5):
print(get_current_ip())
renew_tor_ip()
time.sleep(5)