Using Python to sign into website, fill in a form, then sign out
Question:
As part of my quest to become better at Python I am now attempting to sign in to a website I frequent, send myself a private message, and then sign out. So far, I’ve managed to sign in (using urllib, cookiejar and urllib2). However, I cannot work out how to fill in the required form to send myself a message.
The form is located at /messages.php?action=send. There’s three things that need to be filled for the message to send: three text fields named name, title and message. Additionally, there is a submit button (named “submit”).
How can I fill in this form and send it?
Answers:
import urllib
import urllib2
name = "name field"
data = {
"name" : name
}
encoded_data = urllib.urlencode(data)
content = urllib2.urlopen("http://www.abc.com/messages.php?action=send",
encoded_data)
print content.readlines()
just replace http://www.abc.com/messages.php?action=send with the url where your form is being submitted
reply to your comment: if the url is the url where your form is located, and you need to do this just for one website, look at the source code of the page and find
<form method="POST" action="some_address.php">
and put this address as parameter for urllib2.urlopen
And you have to realise what submit button does.
It just send a Http request to the url defined by action in the form.
So what you do is to simulate this request with urllib2
You want the mechanize library. This lets you easily automate the process of browsing websites and submitting forms/following links. The site I’ve linked to has quite good examples and documentation.
Try to work out the requests that are made (e.g. using the Chrome web developer tool or with Firefox/Firebug) and imitate the POST request containing the desired form data.
In addition to the great mechanize library mentioned by Andrew, in case I’d also suggest you use BeautifulSoup to parse the HTML.
If you don’t want to use mechanize but still want an easy, clean solution to create HTTP requests, I recommend the excellend requests module.
You can use mechanize to work easily with this. This will ease your work of submitting the form. Don’t forget to check with the parameters like name, title, message by seeing the source code of the html form.
import mechanize
br = mechanize.Browser()
br.open("http://mywebsite.com/messages.php?action=send")
br.select_form(nr=0)
br.form['name'] = 'Enter your Name'
br.form['title'] = 'Enter your Title'
br.form['message'] = 'Enter your message'
req = br.submit()
To post data to webpage, use cURL something like this,
curl -d Name="Shrimant" -d title="Hello world" -d message="Hello, how are you" -d Form_Submit="Send" http://www.example.com/messages.php?action=send
The “-d” option tells cURL that the next item is some data to be sent to the server at http://www.example.com/messages.php?action=send
As part of my quest to become better at Python I am now attempting to sign in to a website I frequent, send myself a private message, and then sign out. So far, I’ve managed to sign in (using urllib, cookiejar and urllib2). However, I cannot work out how to fill in the required form to send myself a message.
The form is located at /messages.php?action=send. There’s three things that need to be filled for the message to send: three text fields named name, title and message. Additionally, there is a submit button (named “submit”).
How can I fill in this form and send it?
import urllib
import urllib2
name = "name field"
data = {
"name" : name
}
encoded_data = urllib.urlencode(data)
content = urllib2.urlopen("http://www.abc.com/messages.php?action=send",
encoded_data)
print content.readlines()
just replace http://www.abc.com/messages.php?action=send with the url where your form is being submitted
reply to your comment: if the url is the url where your form is located, and you need to do this just for one website, look at the source code of the page and find
<form method="POST" action="some_address.php">
and put this address as parameter for urllib2.urlopen
And you have to realise what submit button does.
It just send a Http request to the url defined by action in the form.
So what you do is to simulate this request with urllib2
You want the mechanize library. This lets you easily automate the process of browsing websites and submitting forms/following links. The site I’ve linked to has quite good examples and documentation.
Try to work out the requests that are made (e.g. using the Chrome web developer tool or with Firefox/Firebug) and imitate the POST request containing the desired form data.
In addition to the great mechanize library mentioned by Andrew, in case I’d also suggest you use BeautifulSoup to parse the HTML.
If you don’t want to use mechanize but still want an easy, clean solution to create HTTP requests, I recommend the excellend requests module.
You can use mechanize to work easily with this. This will ease your work of submitting the form. Don’t forget to check with the parameters like name, title, message by seeing the source code of the html form.
import mechanize
br = mechanize.Browser()
br.open("http://mywebsite.com/messages.php?action=send")
br.select_form(nr=0)
br.form['name'] = 'Enter your Name'
br.form['title'] = 'Enter your Title'
br.form['message'] = 'Enter your message'
req = br.submit()
To post data to webpage, use cURL something like this,
curl -d Name="Shrimant" -d title="Hello world" -d message="Hello, how are you" -d Form_Submit="Send" http://www.example.com/messages.php?action=send
The “-d” option tells cURL that the next item is some data to be sent to the server at http://www.example.com/messages.php?action=send